product-enrichment

Product Enrichment Node.js Express Cloud Run service which loads product data from BigQuery to Data Store on a daily schedule with a Cloud Endpoints frontend REST API which serves enriched product attributes from Google Cloud Firestore.

Table of Contents

• Description
• Setup
• NPM Package Manager
• HTTPS Request
• Traffic Routing
• Traffic Rate-=limiting
• Endpoint Security
• Test HTTPS Request
  • Authentication
  • Frontend
  • Backend
• Artillery Load Test
  • Config Section
  • Scenarios Section
  • Multi-environment Example
• References

Description

temp removed jest coverageThreshold from package.json because functional test are failing because Bynder has blocked our access to the API.

"coverageThreshold": { "global": { "functions": 65, "branches": 75, "lines": 85, "statements": 85 } },

Product Enrichment Node.js Express Cloud Run service which serves enriched product data to yeti.com and yeti.ca.

Google Cloud Endpoints

• /bynder-webhook - HTTP POST request handler used by the Bynder Application to enrich the Asset media properties.
• /docs - HTTP GET request handler for the Product Enrichment JavaScript API documentation site.
• /health - HTTP GET request handler for the Product Enrichment health check endpoint which returns "HEALTHY".
• /product - HTTP GET request handler which serves product data on demand from Data Store.

Segment.io destination functions which leverage the identify endpoint to perform Personas Identity Resolution and create a unified view of the user across devices, apps, and unique identifiers.

• onIdentity - Handle identify event.
• onPage - Handle page event connections.
• onTrack - Handle track event.

Highlights

• Supports existing data — no additional code or set up required
• Supports all channels — stitches web + mobile + server + third party interactions into the same user
• Supports anonymous identity stitching — by using merging child sessions into parent sessions
• Supports user:account relationships - for b2b companies, generates a graph of relationships between users and accounts
• Realtime - merges realtime data streams, tested at 50,000 resolutions/second with a P95 resolve duration of 7ms

Technical Highlights

• Supports custom external IDs - bring your own external IDs
• Customizable ID Rules — allows you to enforce uniqueness on select external IDs and customize which external IDs and sources cause associations
• Merge Protection - automatically detects and solves identity issues such as non-unique anonymous IDs and the library problem using our priority trust algorithm
• Maintains persistent ID - multiple external IDs get matched to one persistent ID.

Setup

Install Node, npm, n and yarn then load bash_util functions and environment variables into memory.

When your machine does NOT have Node or npm installed.

install node, npm and "n" Node version manager

# install node, npm and "n" Node version manager CLI (i.e. switching between 10, 12, ...)
curl -L https://raw.githubusercontent.com/tj/n/master/bin/n -o n
bash n lts
# Now node and npm are available

When you already have a version (any version) of Node and npm installed.

install yarn Node package manager

# install yarn package manager
npm install -g yarn

# install "n" Node version manager CLI (i.e. switching between 10, 12, ...)
# skip this step if you installed Node and npm using the curl command above
yarn global add n

# install Node 12 and set as default version
sudo n 12
node -v

This project includes the following git subrepos:

• bash-utils: Collection of utility methods and functions used to improve productivity and reduce complexity. Open a terminal and execute ./bash_utils.sh -h to see all the functions that are available along with usage documentation.
• pipeline:

Open a terminal session and execute the script below to clone this repository and load the bash_util functions into memory.

git clone git@gitlab.com:yeti-coolers/dev/terraform/gcp-project-ds.git;

# load the bash_util functions into memory
source bash_util.sh env_configure;

NPM Package Manager

NPM is a package manager for your code. It allows you to use and share code with other developers from around the world.

Node.js distributes code via package (sometimes referred to as modules). A package includes .js JavaScript modules, and a package.json file which describes the package.

• npm Docs
• npm cli commands

# install just npm packages configured in package.json "dependencies" (NOTE: devDependencies are NOT installed)
npm install --production=true;
# install all npm packages configured in package.json including "devDependencies"
npm install --production=false;
# install project dependencies with a clean slate
# meant to be used in automated environments such as test platforms, continuous integration,
# and deployment -- or any situation where you want to make sure dependencies match package.json.
# essentially deletes and recreate node_modules directory
npm ci --production=false;

# install the latest version of a package and any packages that it depends on
# this will also update your package.json and your package-lock.json
# so that other developers working on the project will get the
# same dependencies as you when they run npm install
#   --save - configures one or more packages as a project dependency (used by application) in package.json
npm install --save [package-name];
#   --save-dev - configures one or more packages as a project devDependencies (used for development) in package.json
npm install --save-dev [package-name];
#   --save-exact - configures one or more exact package versions as a project dependency (used by application) in package.json
npm install --save-exact [package-name];
#   --save-optional - configures one or more packages as a project optionalDependencies (can be excluded when not available)
npm install --save-optional [package-name];
#   --save-peer - configures one or more packages as a peerDependencies (required by project peer) in package.json
npm install --save-peer [package-name];

# Running npm uninstall will
#   - remove package from project dependencies
#   - update your package.json
#   - update your package-lock.json file
npm uninstall [package-name];

# audit installed packages for vulnerabilities
npm audit;
# automatically fix vulnerabilities when possible
npm audit fix;

# install npm-check-updates
npx npm-check-updates;

# use npm-check-updates to check package.json dependencies for more recent version.
ncu;

# use npm-check-updates to upgrade dependency versions in package.json to the latest versions
ncu -u;

# Run npm Scripts defined in package.json "scripts" element 

# scan code for syntax issues and problems
npm run lint;
# automatically fix syntax problems when possible
npm run lint:fix;

# run project functional test and unit test
npm run test;
# run project functional test and unit test with verbose logging
npm run test:debug;

# run project functional test just for the common.js module
npm run test:functional:common:debug;
# run project unit test with verbose logging
npm run test:debug;

# run project unit test
npm run test;
# run project unit test with verbose logging
npm run test:debug;

HTTPS Request

Get enriched Product data from Google Cloud Firestore. You can send HTTPS requests from anything able to make HTTPS requests to trigger a Cloud Run-hosted service. Note that all Cloud Run services have a stable HTTPS URL.

GET|POST https://product-enrichment.dev.yeticoolers.tech/

Traffic Routing

The Product Enrichment API uses a VPC Serverless Access connector to route response traffic and outgoing request traffic through Google Cloud NAT which is allocated two static external IPs.

Serverless VPC Access is based on a resource called a connector. A connector handles traffic between your serverless environment and your VPC network. When you create a connector in your Google Cloud project, you attach it to a specific VPC network and region. You can then configure your serverless services to use the connector for outbound network traffic.

A Serverless VPC Access connector consists of connector instances. Serverless VPC Access automatically provisions connector instances depending on the amount of traffic sent through the connector, subject to the min-instances and max-instances settings. Connector instances only scale out and do not scale in. Connector instances can use one of several machine types. Larger machine types provide more throughput.

Serverless VPC Access network tags let you refer to VPC connectors in firewall rules and routes.

Every Serverless VPC Access connector automatically receives two network tags (sometimes called instance tags):

• Universal network tag: vpc-connector Applies to all existing connectors and any connectors made in the future.
• Unique network tag: vpc-connector-REGION-CONNECTOR_NAME Applies to the connector [CONNECTOR_NAME] in [REGION].

These network tags cannot be deleted. New network tags cannot be added.

All requests and responses between the serverless environments and the resources in the VPC network travel internally.

Requests sent to external IP addresses still travel through the internet and do not use the Serverless VPC Access connector.

The following gcloud run deploy arguments are set when deploying the Product Enrichment frontend.

• --vpc-connector [CONNECTOR_PATH] - configures the VPC Serverless Access connector for the Cloud Run app to use.
• --vpc-egress all-traffic configures all outbound traffic from the Cloud Run app to be routed through the VPC Serverless Access connector.

Traffic Rate-limiting

The Bynder Webhook API Endpoint /bynder-webhook uses request slow down and rate-limiting to ensure the Bynder API rate limit of 4500 request per 5 minute time frame is not exceeded.

Request Slow Down

express-slow-down middleware for Express is used to slow down responses rather than blocking them outright. Global slow down is configured using the extensible service proxy IP in order to calculate slow down based on all request.

Response rate will be slowed down after a requestor reaches 4000 request in a 5-minute time frame by adding 500ms of delay per request above 4000.

A req.slowDown property is added to all requests with the following fields:

• limit: The options.delayAfter value (defaults to 1)
• current: The number of requests in the current window
• remaining: The number of requests remaining before rate-limiting begins
• resetTime: When the window will reset and current will return to 0, and remaining will return to limit (in milliseconds since epoch - compare to Date.now()). Note: this field depends on store support.
• delay: Amount of delay imposed on current request (milliseconds)

The Express Rate Limit middleware uses the following configuration:

• delayAfter: max number of connections during windowMs before starting to delay responses. Number or function that returns a number. Defaults to 1. Set to 0 to disable delaying.
• delayMs: milliseconds - how long to delay the response, multiplied by (number of recent hits - delayAfter). Defaults to 1000 (1 second). Set to 0 to disable delaying.
• maxDelayMs: milliseconds - maximum value for delayMs after many consecutive attempts, that is, after the n-th request, the delay will be always maxDelayMs. Important when your application is running behind a load balancer or reverse proxy that has a request timeout. Defaults to Infinity.
• windowMs: milliseconds - how long to keep records of requests in memory. Defaults to 60000 (1 minute).

const bynderApiSlowDown = slowDown({
  delayAfter: 4000, // allow 4000 requests per 5 minutes, then...
  delayMs: 500, // begin adding 500ms of delay per request above 4000
  // request # 4001 is delayed by  500ms
  // request # 4002 is delayed by 1000ms
  // request # 4003 is delayed by 1500ms ...
  maxDelayMs: 5 * 60000 + 500, // 60000 ms = 1 minute - maximum delayMs of 5 min 500 ms
  windowMs: 5 * 60000, // 60000 ms = 1 minute
});

Rate-limiting

express-rate-limit middleware for Express is used to implement rate-limiting for the /bynder-webhook endpoint. Global rate-limiting is configured using the extensible service proxy IP in order to rate-limit based on all request.

The Express Rate Limit middleware uses the following configuration:

• legacyHeaders: Whether to send the legacy rate limit headers for the limit (X-RateLimit-Limit), current usage (X-RateLimit-Remaining) and reset time (if the store provides it) (X-RateLimit-Reset) on all responses. If set to true, the middleware also sends the Retry-After header on all blocked requests.
• max: The maximum number of connections to allow during the window before rate limiting the client.
• message: The response body to send back when a client is rate limited.
• standardHeaders: Whether to enable support for headers conforming to the ratelimit standardization draft adopted by the IETF (RateLimit-Limit, RateLimit-Remaining, and, if the store supports it, RateLimit-Reset). If set to true, the middleware also sends the Retry-After header on all blocked requests. May be used in conjunction with, or instead of the legacyHeaders option.
• statusCode: The HTTP status code to send back when a client is rate limited. Defaults to 429 (HTTP 429 Too Many Requests - RFC 6585).
• windowMs: Time frame for which requests are checked/remembered. Also used in the Retry-After header when the limit is reached.

const bynderApiLimiter = rateLimit({
  legacyHeaders: false, // Disable the `X-RateLimit-*` headers
  max: 4500, // Limit requests to 4500 per `window` (here, per five-minute time frame) from a single IP address.
  message: `Request IP has exceeded the Bynder API request limit of 4500 request per 5 minute time frame.
  Request IP is forbidden from submitting bynder-webhook request for the next 5 minutes.`,
  standardHeaders: true, // Return rate limit info in the `RateLimit-*` headers
  windowMs: 5 * 60000, // 60000 ms = 1 minute
});

Endpoint Security

SSL/TLS is required for the Cloud Run API Endpoint to ensure that data cannot be read in transmission. To successfully submit a GET request the requester must supply a pre-shared secret in the JSON request body and set an X-FS-Signature Header populated with the SHA256 Hash-based Message Authentication Code (HMAC) signature of the JSON request body.

API KEY Deploying an Open API config on Cloud Endpoint creates a new Google API service. console.cloud.google.com -> APIs & Services -> Library -> product-enrichment

The API service name is configured in info.title in the Open API Service configuration openapi-template.yaml. Use the Google Cloud web console UI to create an API Key for the product-enrichment Cloud Endpoints Service: console.cloud.google.com -> API & Services -> Credentials -> CREATE CREDENTIALS -> API key

Best practice is to restrict the API key. Edit the key (click on the pencil), under API restrictions, click on Restrict key and, in the dropdown list, only check your Cloud Endpoints Service API product-enrichment set in openapi-template.yaml info.title.

To enable securing an endpoint path with an API Key, create a EDW_PRODUCT_ENRICHMENT_API_KEY Gitlab Software Development Environment Group (dev/test/stage/prod) CI/CD Variable populated with the API key and add the following configuration to openapi-template.yaml globally or on each path:

security:
- api_key: []
# security can be disabled for all paths by setting
# x-google-allow: all

Test HTTPS Request

The scripts below can be used to verify connectivity and test the Node.js Express application Cloud Endpoints.

Authentication

# shellcheck disable=SC2034
# load bash utility functions and project specific environment variables into memory
source bash_util.sh env_configure;
# set project specific environment variables
_render_templates 'all';

# generate id_token used to authorize calling the API
ACCOUNT="${GIT_PROJECT_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com";
AUDIENCES="https://${BACKEND_SERVICE_DOMAIN}";
ID_TOKEN=$(gcloud auth print-identity-token --impersonate-service-account="${ACCOUNT}" --audiences="${AUDIENCES}" --include-email);

Backend

Node.js Express Service https://product-enrichment-backend.dev.yeticoolers.tech request.

GET Product request

# requires ID_TOKEN authorization above
curl --ssl -v \
  -H "Authorization: Bearer ${ID_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://${BACKEND_SERVICE_DOMAIN}${BASE_PATH}/product?product_id=70000000048&site_id=yeti.com&key=${API_KEY}";

POST Bynder Webhook Notification

# requires ID_TOKEN authorization above
MESSAGE_ID=$(uuid);
curl -X POST --ssl -v -d '{"Type":"Notification","MessageId":"f3560e5b-79d8-5ae0-a781-539ff4a14c4d","TopicArn":"arn:aws:sns:eu-central-1:558288162421:yeti-05DCBBB3-7370-410B-A8AA6FBF550B7B2C-prod","Subject":"asset_bank.media.create","Message":"{\"media_id\": \"A77DAFEC-5887-43B1-910D952D6CD1DB20\", \"media\": {\"property_ProductSKU\": [\"70000001153\", \"18060131072\"], \"userCreated\": \"Jennifer Allen\", \"limited\": 0, \"watermarked\": 0, \"mediaItems\": [{\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"INT_WEB_ANGLE-2023_YetiXDolomites__R__1105.png\", \"id\": \"014E8F2A-4002-40EF-B267E33F8DE55DED\", \"size\": 1532322, \"height\": 850, \"width\": 920, \"focusPoint\": {\"x\": 460.0, \"y\": 425.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"crm_banner-2023_YetiXDolomites__R__1105.png\", \"id\": \"04204701-564B-4405-A9CADC527F336FBA\", \"size\": 501020, \"height\": 400, \"width\": 600, \"focusPoint\": {\"x\": 300.0, \"y\": 200.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"Social_Media 1080x1920-2023_YetiXDolomites__R__1105.png\", \"id\": \"26A52207-CF79-42DE-9E41C9B061BE1734\", \"size\": 3665169, \"height\": 1920, \"width\": 1080, \"focusPoint\": {\"x\": 540.0, \"y\": 960.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"INT_WEB-2023_YetiXDolomites__R__1105.jpg\", \"id\": \"38C02343-9381-411B-B368C81FC1A0DEFA\", \"size\": 463000, \"height\": 1024, \"width\": 1680, \"focusPoint\": {\"x\": 840.0, \"y\": 512.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"W-2023_YetiXDolomites__R__1105.png\", \"id\": \"43023DA2-8B24-4F87-9478B65DE292950D\", \"size\": 6649621, \"height\": 1920, \"width\": 1920, \"focusPoint\": {\"x\": 960.0, \"y\": 960.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"INT_B2B-2023_YetiXDolomites__R__1105.png\", \"id\": \"44B5996B-F3D2-467F-9304F182B04C89E5\", \"size\": 720385, \"height\": 600, \"width\": 600, \"focusPoint\": {\"x\": 300.0, \"y\": 300.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"crm_grid-2023_YetiXDolomites__R__1105.png\", \"id\": \"7536B104-2491-49DC-BC85FE7455955169\", \"size\": 187856, \"height\": 300, \"width\": 300, \"focusPoint\": {\"x\": 150.0, \"y\": 150.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"2023_YetiXDolomites__R__1105.jpg\", \"id\": \"C1EDD9F6-81DF-487E-B40362A142E42D5A\", \"size\": 36415363, \"height\": 5464, \"width\": 8192, \"focusPoint\": {\"x\": 4096.0, \"y\": 2732.0}}, {\"version\": 0, \"active\": 0, \"dateCreated\": \"2023-11-07T19:08:20Z\", \"fileName\": \"2023_YetiXDolomites__R__1105.tif\", \"id\": \"C7313C24-0504-45FE-8A389D7E12B9E04B\", \"size\": 268603304, \"height\": 5464, \"width\": 8192, \"focusPoint\": {\"x\": 4096.0, \"y\": 2732.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"Social_Media_1920x1080_PNG-2023_YetiXDolomites__R__1105.png\", \"id\": \"DA98A797-57B7-4BEE-BD9D8C7E6AB60BAD\", \"size\": 4120952, \"height\": 1080, \"width\": 1920, \"focusPoint\": {\"x\": 960.0, \"y\": 540.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"crm_hero-2023_YetiXDolomites__R__1105.png\", \"id\": \"DC922890-670D-40E2-A4A1CAF5799F2640\", \"size\": 929159, \"height\": 800, \"width\": 600, \"focusPoint\": {\"x\": 300.0, \"y\": 400.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"Social Media 1200x675-2023_YetiXDolomites__R__1105.png\", \"id\": \"E7B52103-E184-428B-B69D66374274DBD2\", \"size\": 1683083, \"height\": 675, \"width\": 1200, \"focusPoint\": {\"x\": 600.0, \"y\": 337.5}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"Social Media 1080x1080-2023_YetiXDolomites__R__1105.png\", \"id\": \"F4F12FB4-E780-48D4-A4113D6E9134D55B\", \"size\": 2224202, \"height\": 1080, \"width\": 1080, \"focusPoint\": {\"x\": 540.0, \"y\": 540.0}}], \"activeOriginalFocusPoint\": {\"x\": 4096.0, \"y\": 2732.0}, \"archive\": 0, \"brandId\": \"A39CE806-4E2C-41D0-8EF95BB1B7EDA300\", \"copyright\": \"2020\", \"datePublished\": \"2023-07-09T02:30:23Z\", \"dateCreated\": \"2023-07-20T22:12:15Z\", \"description\": \"\", \"dateModified\": \"2023-11-07T19:08:36Z\", \"extension\": [\"jpg\"], \"fileSize\": 36415363, \"height\": 5464, \"id\": \"A77DAFEC-5887-43B1-910D952D6CD1DB20\", \"idHash\": \"6193b3b1ac14676e\", \"isPublic\": 0, \"name\": \"2023_YetiXDolomites__R__1105\", \"orientation\": \"landscape\", \"propertyOptions\": [\"03E3023C-3145-4668-AE950C2B03DEDBA2\", \"1A3AA0DF-AB7C-4851-881391955423EB9B\", \"21D936BF-A153-4803-BBF0301FFF2E7318\", \"25616947-08C8-4416-B8C07248B22BFDE5\", \"28055C24-6845-4449-966BD9B13177A3EF\", \"2BDC886F-FBB1-4C83-97E5F9BDB0BD9EE3\", \"3B78F056-53C6-4DDF-906BC4DB0CF210DF\", \"3D7036DD-5D9E-472A-AEE778DF4CCB63A2\", \"42387F0E-280C-4C44-8182773F15C71DA7\", \"45302EBC-7044-43DD-A12BAB17B3658669\", \"480C0795-6ADF-4212-8AABBF5DFE9E676E\", \"48653538-2542-4338-8B7766096DA53A72\", \"5C03E489-5EC1-4A3E-B2D16323E4E82945\", \"68A3B36D-77DE-43A5-B83B915E95409286\", \"850AEC93-17AC-47A3-9597F64E0EBC3E8E\", \"89092E35-B9DB-4E5B-B9C0DEE5CCAF93C5\", \"909082A0-9B2E-4E7D-80B391815B197373\", \"ECCED462-B44A-4161-9B10BED9DFAFD33D\", \"F494C7F0-54D4-4AA0-90D6E230D3D2F0E9\", \"F4D64451-1793-4B1E-AA39C7100681F316\"], \"tags\": [], \"type\": \"image\", \"width\": 8192, \"property_KnownUses\": \"Dispatch 2H23\", \"property_ExpirationDate\": \"2025-11-07T00:00:00Z\", \"property_contentType\": [\"lifestyle\"], \"property_AssetLevel\": [\"dispatchembargo\"], \"property_Animal\": [\"noAnimalShown\"], \"property_ExpirationYear\": [\"2025\"], \"property_Asset_Type\": [\"Content_Library\"], \"property_PhotographerName\": [\"travisrummel\"], \"property_AssetType\": [\"Images\"], \"property_Outdoors\": [\"Hiking\"], \"property_usagestatus\": [\"fulluse\"], \"property_ShootName\": [\"Travis_Rummel_-_2023_-_Dolomites_Hut_to_Hut_Dispatch\"], \"property_Year\": [\"2023\"], \"property_selectsAvailable\": [\"Yes\"], \"property_photoshootproducer\": [\"Courtney_Katz\"], \"property_assetFormat\": [\"originalPhotography\"], \"property_AssetSubtype\": [\"Lifestyleimages\"], \"property_Pursuit\": [\"Outdoors\"], \"property_Term\": [\"2Year\"], \"property_High_Res_File\": [\"Yes\"], \"property_ExpirationMonth\": [\"November\"], \"property_ProductName\": [\"No_Product_Shown\"]}}","Timestamp":"2023-11-07T19:08:41.905Z","SignatureVersion":"1","Signature":"CJIYBKR8jiIBh/PfeBrwwqdXTT/b/XZ7Ds6mIRfw+Dya6FUt2IwKXF4GU55u60TIcChoxjDJxp2XDg5aKGH2KVc0u8LuPQFJDvL87XZks3kcGYt651ETqEbgb7l7NvEFc7TZorRTe1YNN8IbbxZV1wIqI0UkYHVd1oww9dTHBx1eVwoOqImW8x1U+Ix1NVY3+Ns6XhE2eWsjdiZwHM4iD35ZCWG++Kk1pD8pW2WUNQSaEkN8eBK59Tnt5Mdtdtxzl58/NUch71ckVK9gzPi9n1H+UtDiIuOhVAzV4ld3BEzemrKENia5UnLnlBUQoiOrhmybexQCTuE7S1Ew0NsfTg==","SigningCertURL":"https://sns.eu-central-1.amazonaws.com/SimpleNotificationService-01d088a6f77103d0fe307c0069e40ed6.pem","UnsubscribeURL":"https://sns.eu-central-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-central-1:558288162421:yeti-05DCBBB3-7370-410B-A8AA6FBF550B7B2C-prod:3cb6215d-f396-4547-8788-2e1162e3cb9c","MessageAttributes":{"traceparent":{"Type":"String","Value":"00-e6df13df77c6a489d9da0b2819d403ef-c4c0330a9c017ae2-01"}}}' \
  -H "Authorization: Bearer ${ID_TOKEN}" \
  -H "Content-Type: application/json" \
  -H "x-amz-sns-message-type: Notification" \
  -H "x-amz-sns-message-id: ${MESSAGE_ID}" \
  https://${BACKEND_SERVICE_DOMAIN}${BASE_PATH}/bynder-webhook;

Frontend

Extensible Service Proxy v2 (ESPv2) https://product-enrichment.dev.yeticoolers.tech request.

GET Product request

# requires ID_TOKEN authorization above
curl --ssl -v \
  -H "Authorization: Bearer ${ID_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://${FRONTEND_SERVICE_DOMAIN}${BASE_PATH}/product?product_id=70000000028&site_id=yeti.com&key=${API_KEY}";

POST Bynder Webhook Notification

# requires ID_TOKEN authorization above
MESSAGE_ID=$(uuid);
curl -X POST --ssl -v -d '{"Type":"Notification","MessageId":"f3560e5b-79d8-5ae0-a781-539ff4a14c4d","TopicArn":"arn:aws:sns:eu-central-1:558288162421:yeti-05DCBBB3-7370-410B-A8AA6FBF550B7B2C-prod","Subject":"asset_bank.media.create","Message":"{\"media_id\": \"A77DAFEC-5887-43B1-910D952D6CD1DB20\", \"media\": {\"property_ProductSKU\": [\"70000001153\", \"18060131072\"], \"userCreated\": \"Jennifer Allen\", \"limited\": 0, \"watermarked\": 0, \"mediaItems\": [{\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"INT_WEB_ANGLE-2023_YetiXDolomites__R__1105.png\", \"id\": \"014E8F2A-4002-40EF-B267E33F8DE55DED\", \"size\": 1532322, \"height\": 850, \"width\": 920, \"focusPoint\": {\"x\": 460.0, \"y\": 425.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"crm_banner-2023_YetiXDolomites__R__1105.png\", \"id\": \"04204701-564B-4405-A9CADC527F336FBA\", \"size\": 501020, \"height\": 400, \"width\": 600, \"focusPoint\": {\"x\": 300.0, \"y\": 200.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"Social_Media 1080x1920-2023_YetiXDolomites__R__1105.png\", \"id\": \"26A52207-CF79-42DE-9E41C9B061BE1734\", \"size\": 3665169, \"height\": 1920, \"width\": 1080, \"focusPoint\": {\"x\": 540.0, \"y\": 960.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"INT_WEB-2023_YetiXDolomites__R__1105.jpg\", \"id\": \"38C02343-9381-411B-B368C81FC1A0DEFA\", \"size\": 463000, \"height\": 1024, \"width\": 1680, \"focusPoint\": {\"x\": 840.0, \"y\": 512.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"W-2023_YetiXDolomites__R__1105.png\", \"id\": \"43023DA2-8B24-4F87-9478B65DE292950D\", \"size\": 6649621, \"height\": 1920, \"width\": 1920, \"focusPoint\": {\"x\": 960.0, \"y\": 960.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"INT_B2B-2023_YetiXDolomites__R__1105.png\", \"id\": \"44B5996B-F3D2-467F-9304F182B04C89E5\", \"size\": 720385, \"height\": 600, \"width\": 600, \"focusPoint\": {\"x\": 300.0, \"y\": 300.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"crm_grid-2023_YetiXDolomites__R__1105.png\", \"id\": \"7536B104-2491-49DC-BC85FE7455955169\", \"size\": 187856, \"height\": 300, \"width\": 300, \"focusPoint\": {\"x\": 150.0, \"y\": 150.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"2023_YetiXDolomites__R__1105.jpg\", \"id\": \"C1EDD9F6-81DF-487E-B40362A142E42D5A\", \"size\": 36415363, \"height\": 5464, \"width\": 8192, \"focusPoint\": {\"x\": 4096.0, \"y\": 2732.0}}, {\"version\": 0, \"active\": 0, \"dateCreated\": \"2023-11-07T19:08:20Z\", \"fileName\": \"2023_YetiXDolomites__R__1105.tif\", \"id\": \"C7313C24-0504-45FE-8A389D7E12B9E04B\", \"size\": 268603304, \"height\": 5464, \"width\": 8192, \"focusPoint\": {\"x\": 4096.0, \"y\": 2732.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"Social_Media_1920x1080_PNG-2023_YetiXDolomites__R__1105.png\", \"id\": \"DA98A797-57B7-4BEE-BD9D8C7E6AB60BAD\", \"size\": 4120952, \"height\": 1080, \"width\": 1920, \"focusPoint\": {\"x\": 960.0, \"y\": 540.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"crm_hero-2023_YetiXDolomites__R__1105.png\", \"id\": \"DC922890-670D-40E2-A4A1CAF5799F2640\", \"size\": 929159, \"height\": 800, \"width\": 600, \"focusPoint\": {\"x\": 300.0, \"y\": 400.0}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"Social Media 1200x675-2023_YetiXDolomites__R__1105.png\", \"id\": \"E7B52103-E184-428B-B69D66374274DBD2\", \"size\": 1683083, \"height\": 675, \"width\": 1200, \"focusPoint\": {\"x\": 600.0, \"y\": 337.5}}, {\"version\": 1, \"active\": 1, \"dateCreated\": \"2023-07-20T22:12:15Z\", \"fileName\": \"Social Media 1080x1080-2023_YetiXDolomites__R__1105.png\", \"id\": \"F4F12FB4-E780-48D4-A4113D6E9134D55B\", \"size\": 2224202, \"height\": 1080, \"width\": 1080, \"focusPoint\": {\"x\": 540.0, \"y\": 540.0}}], \"activeOriginalFocusPoint\": {\"x\": 4096.0, \"y\": 2732.0}, \"archive\": 0, \"brandId\": \"A39CE806-4E2C-41D0-8EF95BB1B7EDA300\", \"copyright\": \"2020\", \"datePublished\": \"2023-07-09T02:30:23Z\", \"dateCreated\": \"2023-07-20T22:12:15Z\", \"description\": \"\", \"dateModified\": \"2023-11-07T19:08:36Z\", \"extension\": [\"jpg\"], \"fileSize\": 36415363, \"height\": 5464, \"id\": \"A77DAFEC-5887-43B1-910D952D6CD1DB20\", \"idHash\": \"6193b3b1ac14676e\", \"isPublic\": 0, \"name\": \"2023_YetiXDolomites__R__1105\", \"orientation\": \"landscape\", \"propertyOptions\": [\"03E3023C-3145-4668-AE950C2B03DEDBA2\", \"1A3AA0DF-AB7C-4851-881391955423EB9B\", \"21D936BF-A153-4803-BBF0301FFF2E7318\", \"25616947-08C8-4416-B8C07248B22BFDE5\", \"28055C24-6845-4449-966BD9B13177A3EF\", \"2BDC886F-FBB1-4C83-97E5F9BDB0BD9EE3\", \"3B78F056-53C6-4DDF-906BC4DB0CF210DF\", \"3D7036DD-5D9E-472A-AEE778DF4CCB63A2\", \"42387F0E-280C-4C44-8182773F15C71DA7\", \"45302EBC-7044-43DD-A12BAB17B3658669\", \"480C0795-6ADF-4212-8AABBF5DFE9E676E\", \"48653538-2542-4338-8B7766096DA53A72\", \"5C03E489-5EC1-4A3E-B2D16323E4E82945\", \"68A3B36D-77DE-43A5-B83B915E95409286\", \"850AEC93-17AC-47A3-9597F64E0EBC3E8E\", \"89092E35-B9DB-4E5B-B9C0DEE5CCAF93C5\", \"909082A0-9B2E-4E7D-80B391815B197373\", \"ECCED462-B44A-4161-9B10BED9DFAFD33D\", \"F494C7F0-54D4-4AA0-90D6E230D3D2F0E9\", \"F4D64451-1793-4B1E-AA39C7100681F316\"], \"tags\": [], \"type\": \"image\", \"width\": 8192, \"property_KnownUses\": \"Dispatch 2H23\", \"property_ExpirationDate\": \"2025-11-07T00:00:00Z\", \"property_contentType\": [\"lifestyle\"], \"property_AssetLevel\": [\"dispatchembargo\"], \"property_Animal\": [\"noAnimalShown\"], \"property_ExpirationYear\": [\"2025\"], \"property_Asset_Type\": [\"Content_Library\"], \"property_PhotographerName\": [\"travisrummel\"], \"property_AssetType\": [\"Images\"], \"property_Outdoors\": [\"Hiking\"], \"property_usagestatus\": [\"fulluse\"], \"property_ShootName\": [\"Travis_Rummel_-_2023_-_Dolomites_Hut_to_Hut_Dispatch\"], \"property_Year\": [\"2023\"], \"property_selectsAvailable\": [\"Yes\"], \"property_photoshootproducer\": [\"Courtney_Katz\"], \"property_assetFormat\": [\"originalPhotography\"], \"property_AssetSubtype\": [\"Lifestyleimages\"], \"property_Pursuit\": [\"Outdoors\"], \"property_Term\": [\"2Year\"], \"property_High_Res_File\": [\"Yes\"], \"property_ExpirationMonth\": [\"November\"], \"property_ProductName\": [\"No_Product_Shown\"]}}","Timestamp":"2023-11-07T19:08:41.905Z","SignatureVersion":"1","Signature":"CJIYBKR8jiIBh/PfeBrwwqdXTT/b/XZ7Ds6mIRfw+Dya6FUt2IwKXF4GU55u60TIcChoxjDJxp2XDg5aKGH2KVc0u8LuPQFJDvL87XZks3kcGYt651ETqEbgb7l7NvEFc7TZorRTe1YNN8IbbxZV1wIqI0UkYHVd1oww9dTHBx1eVwoOqImW8x1U+Ix1NVY3+Ns6XhE2eWsjdiZwHM4iD35ZCWG++Kk1pD8pW2WUNQSaEkN8eBK59Tnt5Mdtdtxzl58/NUch71ckVK9gzPi9n1H+UtDiIuOhVAzV4ld3BEzemrKENia5UnLnlBUQoiOrhmybexQCTuE7S1Ew0NsfTg==","SigningCertURL":"https://sns.eu-central-1.amazonaws.com/SimpleNotificationService-01d088a6f77103d0fe307c0069e40ed6.pem","UnsubscribeURL":"https://sns.eu-central-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-central-1:558288162421:yeti-05DCBBB3-7370-410B-A8AA6FBF550B7B2C-prod:3cb6215d-f396-4547-8788-2e1162e3cb9c","MessageAttributes":{"traceparent":{"Type":"String","Value":"00-e6df13df77c6a489d9da0b2819d403ef-c4c0330a9c017ae2-01"}}}' \
  -H "Authorization: Bearer ${ID_TOKEN}" \
  -H "Content-Type: application/json" \
  -H "x-amz-sns-message-type: Notification" \
  -H "x-amz-sns-message-id: ${MESSAGE_ID}" \
  https://${FRONTEND_SERVICE_DOMAIN}${BASE_PATH}/bynder-webhook;

docs

curl --ssl -v "https://${FRONTEND_SERVICE_DOMAIN}${BASE_PATH}/docs";

Artillery Load Test

An Artillery test script is composed of two sections: config and scenarios.

• config: section defines the target (the hostname or IP address of the system under test), the load progression, and protocol-specific settings such as HTTP response timeouts or Socket.io transport options.
• scenarios: section contains definitions for one or more scenarios for the virtual users (VUs) that Artillery will create.

Config Section

• target - the URI of the application under test. For an HTTP application, the base URL for all requests, e.g. http://myapp.staging.local. For a WebSocket server it would be the hostname (and optionally the port) of the server, e.g. ws://127.0.0.1.
• environments - specify a list of environments, and associated target URLs; see below
• phases - specify the duration of the test and frequency of requests; phases is an array of phase definitions that Artillery goes through sequentially. Four kinds of phases are supported:
  • A phase with a duration and a constant arrival rate of a number of new VUs per second.
  • A linear ramp up phase where the number of new arrivals increases linearly over the duration of the phase.
  • A phase which generates a fixed count of new arrivals over a period of time.
  • A pause phase which generates no new VUs for a duration of time.
• payload - used for importing variables from a CSV file; see below
• variables - set variables inline rather than loading them from an external CSV file
• defaults - set default headers that will apply to all HTTP requests
• tls - configure how Artillery handles self-signed certificates. See HTTP Reference
• ensure - set success conditions for latency or error rates; useful for CI/CD

Using dynamic values in config

Values can be set dynamically via environment variables which are available under $processEnvironment template variable.

config:
  target: https://product-enrichment.dev.yeticoolers.tech
  phases:
    - duration: 600
      arrivalRate: 10
  defaults:
    headers:
      x-api-key: "{{ $processEnvironment.SERVICE_API_KEY }}"
scenarios:
  - flow:
    - log: "Artillery Load Test Environment: {{ $environment }}"
    - get:
        url: "/v1/product"

Load Phases

A load phase defines how many new virtual users (VUs) will be generated in a time period. For example, a typical performance test will have a gentle warm up phase, followed by a ramp up phase which is then followed by a maximum load for a duration of time.

Total number of VUs can be capped with the maxVusers option.

Environments

Define environment specific test using config.environments. The name of the current environment (if set) is available in the $environment variable. Choose an environment on the command line with the -e flag.

artillery run -e prod artillery-load-test.yml

Setting success conditions with ensure

When running Artillery in CI/CD pipelines, it can be useful to have Artillery exit with a non-zero code when a condition is not met.

• min, max, median, p95, and p99: Check aggregate latency, and make Artillery exit with a non-zero exit code if it is over the configured value.
• maxErrorRate: The error rate is defined as the ratio of virtual users that didn't complete their scenarios successfully to the total number of virtual users created during the test.

Scenarios Section

The scenarios section is where one or more virtual user scenarios are defined.

A scenario is a sequence of steps that will be run sequentially which represents a typical sequence of requests or messages sent by a user of an application.

A scenario definition is an object which must contain a flow attribute and may contain a number of other attributes.

• flow: a "flow" is an array of operations that a virtual user performs, e.g. GET and POST requests for an HTTP-based application.
• name: allows to assign a descriptive name to a scenario, e.g. "search for a product and get its details"
• weight: allows for the probability of a scenario being picked by a new virtual user to be "weighed" relative to other scenarios

Some Artillery engines will also support other scenario attributes, e.g. the HTTP engine allows for scenario-level hooks to be defined.

Scenario weights

Weights allow you to specify that some scenarios should be picked more often than others. If you have three scenarios with weights 1, 2, and 5, the scenario with the weight of 2 is twice as likely to be picked as the one with the weight of 1, and 2.5 times less likely than the one with weight of 5. Or in terms of probabilities:

• scenario 1: 1/8 = 12.5% probability of being picked
• scenario 2: 2/8 = 25% probability
• scenario 3: 5/8 = 62.5% probability

Weights are optional, and if not specified are set to 1 (i.e. each scenario is equally likely to be picked).

Multi-environment Example

• load product_id values and site_id values from csv data
• default (dev):
  • ramp up arrival rate from 10 to 50 over 2 minutes with no more than 50 concurrent VUs, followed by 10 minutes at 50 arrivals per second.
  • exit with a non-zero if the total error rate exceeded 5%
  • check the aggregate p95 latency is 200ms or less
• prod:
  • double the load, ramp up arrival rate from 20 to 100 over 2 minutes with no more than 100 concurrent VUs, followed by 10 minutes at 100 arrivals per second.
  • exit with a non-zero if the total error rate exceeded 1%
  • check the aggregate p99 latency is 200ms or less

config:
  target: "https://product-enrichment.dev.yeticoolers.tech"
  defaults:
    headers:
      X-API-Key: "{{ $processEnvironment.API_KEY }}"
  payload:
    # load product_id values and site_id values from csv data
    # path is relative to the location of the artillery-load-test.yml configuration file
    # BigQuery SQL used to create load-test-data.csv
    # SELECT product_id,
    #   CASE WHEN sales_org = '1100' THEN 'yeti.com'
    #   ELSE 'yeti.ca' END AS site_id
    # FROM `yeti-dev-edw.edw.products`
    # WHERE sales_org IN('1100', '1500')
    #   AND distribution_channel = '10';
    path: "load-test-data.csv"
    fields:
      - "product_id"
      - "site_id"
  phases:
    - name: "Warm up the application"
      duration: 120
      arrivalRate: 10
      maxVusers: 50
      rampTo: 50
    - name: "Sustained max load"
      duration: 600
      arrivalRate: 50
  ensure:
    maxErrorRate: 5
    p95: 200
  environments:
    prod:
      target: "https://product-enrichment.prod.yeticoolers.tech"
      phases:
        - name: "Warm up the application"
          duration: 120
          arrivalRate: 20
          maxVusers: 100
          rampTo: 100
        - name: "Sustained max load"
          duration: 600
          arrivalRate: 100
      ensure:
        maxErrorRate: 1
        p99: 200
scenarios:
  - flow:
    - get:
        url: "{{ basePath }}/product"
        qs:
          product_id: "{{ product_id }}"
          site_id: "{{ site_id }}"

Initial Load Test Results

All virtual users finished
Summary report @ 15:09:51(-0500) 2020-07-28
  Scenarios launched:  10265
  Scenarios completed: 10265
  Requests completed:  10265
  Mean response/sec: 56.76
  Response time (msec):
    min: 148.9
    max: 4323.9
    median: 321.4
    p95: 730.5
    p99: 1135.8
  Scenario counts:
    0: 10265 (100%)
  Codes:
    200: 10265

References

• Configuring Serverless VPC Access
• gcloud run deploy
• Routing Cloud Run and Cloud Functions egress traffic through a static external IP address
• Serverless VPC Access