Daniel Hundt

We used Amazon S3 for shared file storage and Amazon CloudFront as CDN in combination with AWS Lambda (and a old custom script) for delivering images. This was a very cost intensive setup which would get more and more expensive with increasing traffic and multiple image dimensions. So we looked for alternatives and found bunny.net

With bunny storage we have a much cheaper alternative to S3. Increasing availability by using multiple storage zones was easy to set up.

Also the CDN is much cheaper in regards to traffic at a competitive (in their own comparison: even better) performance.

The image transformation comes at a low fixed price without any additional maintenance work on our side and allows delivery of optimized images based on the clients ability.

We are happy with the results, although especially the documentation could be a lot better and the image transformation options are currently only very basic. There is e.g. no "crop to interesting area" option (like with e.g. Cloudinary). Currently we use Thumbor to get around this.

When analyzing our access log, we saw some attack patterns. We wrote some block rules to filter the most common ones. These are the ones, that fit our application. Since we use Symfony with speaking URLs, eg. no valid URL should contain a ".php" ending. This rule alone prevents most of the traffic reaching deeper into the stack. Also blocking of dot-files (common configs) is an additional safety net. Since we analyze our access log on a regular base, we can add additional rules if needed.

map $request_uri $isBlocked {
  default 0;
  # still allow .well-known
  ~^/\.well-known 0;

  # wordpress attack
  ~*.*wp-(admin|login|content).* 1;
  ~*.*wordpress.* 1;

  # other common attacks
  ~*.*(phpmyadmin|pma|myadmin)([^a-zA-Z0-9\.\-].*)?$ 1;
  ~*.*(typo3|joomla|drupal|contao).* 1;
  ~*.*(setup|install)\.php.* 1;
  ~*.*(cgi-|cgibin).* 1;
  ~*.*/(webtools|ws|storage|htbin)/.* 1;

  # file endings: x.exe, x.exe.foo, x.exe/foo; but not x.exefoo
  ~*.*\.(exe|cmd|asp|cgi|pl|php|sql|py|cfg|yaml|yml|dat|log|lock)([^a-zA-Z0-9\.\-].*)?$ 1;

  # like "php://filter/convert.iconv..." or other like "ftp://" in path
  ~*.://.* 1;

  # parent directory "../"
  ~*.*\.\./.* 1;
  # dot files like, .env, .git/config
  ~*.*/\..* 1;

  # sql injections
  ~*.*union.*select.* 1;
  ~*.*('|")(.*)(drop|insert|md5|select|union).* 1;

  # script tag injection
  ~*.*(<|%3C).*script.* 1;

  # evil methods
  ~*.*(eval|xor|sysdate|sleep|now|chr|delay)\(.* 1;

  # base64 en-/decoding
  ~*.*base64(_)?(en|de)code.* 1;

  # php super globals
  ~*.*(GLOBALS|ENV|REQUEST|SESSION|SERVER)(=|\[|\%).* 1;
}

map $http_referer $isInvalidReferer {
  default 1;

  # empty is fine
  "" 0;
  "-" 0;

  # should contain valid URL including protocol
  ~^http(s)?://.+ 0;
}

server {
  # ...

  # prevent communication of current nginx version
  server_tokens off;

  if ($isBlocked) {
    return 404;
  }
  if ($isInvalidReferer) {
    return 400;
  }

  # location ...
}

Besides this, attacks are reported to the providers abuse contact. If attacks don't stop we use an additional rule set for blocking IP-ranges.

We plan to migrate our continous integration platform from Drone to Woodpecker CI, since the open source version of drone is no longer maintained. Alternatives were Jenkins, Bitbucket Pipelines, Concours CI or a migration to GitLab. Woodpecker CI as fork looked like the easiest way to migrate. The migration is currently evaluated. Results may follow later.

We use Redis as common in-memory cache for our distributed php processes. Since it also provides message-queue functions and was already in our stack, we didn't use an alternative like RabbitMQ for async handling. We have a multi-instance setup (configured via Ansible) on our maschines. High-Availaibility is configured via Redis Sentinel and HAProxy. The HAProxy-HAProxy setup is also responsible for SSL encryption. We could not use twemproxy since not all commands our application uses are supported.

We were not happy with the price/performance of Amazon RDS (Aurora) and wanted a high availability setup. In order to keep the price low and be compatible with the MySQL API, we decided to try Percona Xtra DB Cluster. Setup was rather complex but could be automated via Ansible. We now run multiple instances (with separate resource limits) on our servers. Since ProxySQL won't support the "LOAD DATA" statement, we used a HAProxy-HAProxy Setup in tcp-mode, which would also give us transparent SSL encryption without changes in the application. For health-check and monitoring we wrote a small express.js application that sends request to all instances and exports in the Prometheus format.