Autoscaling for AWS EMR Clusters

Allow EMR Clusters to Auto Scale based on cluster usage, allowing our applications to be more reactive and cost-efficient.

Estimated Impact

  • Better performance and reaction to load spikes.
  • Cost savings on EMR resource usage.

Measurement

  • Measurement can be done by comparing the overall number of deployed instances before/after Autoscaling is enabled everywhere possible.
  • Also, it might be possible that by enabling AutoScaling, some EMR Instance Types may be changed to less powerful ones.

What we know so far

About EMR Auto-scaling

  • EMR allows auto-scaling only on TASK node types. You can set any instance type, which is going to be used for new servers.
  • You can set the minimum and maximum instances there can be. (I have gone with 0 and 6 respectively)
  • You can set the thresholds for scaling out and in on instance count. Available metrics are the same as CloudWatch.
  • You need to specify an Autoscaling role. There’s a default AWS creates for you: `EMR_AutoScaling_DefaultRole`.
  • All this can be configured via CloudFormation using the InstanceGroupConfig type.
  • There is an issue where a cluster/stack with this configuration will fail to delete the regular way. More info HERE.
"FeedingAutoScale": {
       "Type" : "AWS::EMR::InstanceGroupConfig",
       "Properties" : {
         "AutoScalingPolicy" : {
           "Constraints": {
             "MaxCapacity": {
               "Ref": "AutoScalingInstanceMax"
             },
             "MinCapacity": {
               "Ref": "AutoScalingInstanceMin"
             }
           },
           "Rules": [
             {
               "Name": "Scale-out",
               "Description": "Scale-out policy",
               "Action": {
                 "SimpleScalingPolicyConfiguration": {
                   "AdjustmentType": "CHANGE_IN_CAPACITY",
                   "ScalingAdjustment": 1,
                   "CoolDown": 300
                 }
               },
               "Trigger": {
                 "CloudWatchAlarmDefinition": {
                   "Dimensions": [
                     {
                       "Key": "JobFlowId",
                       "Value": {
                         "Ref": "FeedingCluster"
                       }
                     }
                   ],
                   "EvaluationPeriods": 1,
                   "Namespace": "AWS/ElasticMapReduce",
                   "Period": 300,
                   "ComparisonOperator": "LESS_THAN",
                   "Statistic": "AVERAGE",
                   "Threshold": {
                     "Ref": "AutoScalingScaleOutYarnMAP"
                   },
                   "Unit": "PERCENT",
                   "MetricName": "YARNMemoryAvailablePercentage"
                 }
               }
             },
             {
               "Name": "Scale-in",
               "Description": "Scale-in policy",
               "Action": {
                 "SimpleScalingPolicyConfiguration": {
                   "AdjustmentType": "CHANGE_IN_CAPACITY",
                   "ScalingAdjustment": -1,
                   "CoolDown": 300
                 }
               },
               "Trigger": {
                 "CloudWatchAlarmDefinition": {
                   "Dimensions": [
                     {
                       "Key": "JobFlowId",
                       "Value": {
                         "Ref": "FeedingCluster"
                       }
                     }
                   ],
                   "EvaluationPeriods": 1,
                   "Namespace": "AWS/ElasticMapReduce",
                   "Period": 300,
                   "ComparisonOperator": "GREATER_THAN",
                   "Statistic": "AVERAGE",
                   "Threshold": {
                     "Ref": "AutoScalingScaleInYarnMAP"
                   },
                   "Unit": "PERCENT",
                   "MetricName": "YARNMemoryAvailablePercentage"
                 }
               }
             }
           ]
         },
         "Name": "AutoScaling TASK",
         "InstanceCount": {
           "Ref": "AutoScalingInstanceCount"
         },
         "InstanceRole" : "TASK",
         "InstanceType": {
           "Ref": "AutoScalingInstanceType"
         },
         "JobFlowId": {
           "Ref": "FeedingCluster"
         }
       }
     }

About Dynamic Allocation

For AutoScaling to work correctly, no fixed set of executors need to be configured. Instead, Spark needs to be able to allocate resources dynamically. This concept has been around for a while. It uses previous job metrics to calculate a dynamic load. There exists two relevant settings for this:

  • `spark.dynamicAllocation.enabled`  – Since Spark 1.5
  • `spark.streaming.dynamicAllocation.enabled`  – Since Spark 2.0.0

As our Feeding application is a Spark streaming one, we need the latter. However, EMR automatically sets `spark.dynamicAllocation.enabled` to true behind doors in an attempt to optimize apps for you. As these two settings are incompatible (They can’t be both enabled), we need to explicitly disable it. More info on Spark settings EMR changes automatically can be found HERE.

Other spark configs that don’t play well with StreamingDynamicAllocation are:

  • `spark.executor.cores`
  • `spark.executor.instances`

Those need to be left out.

On upgrading EMR (and Spark) versions

For Spark Streaming Dynamic Allocation, it is recommended to use Spark version > 2.0.1. Currently, our setup uses EMR release version 5.0.0, which comes with Spark 2.0.0. An EMR release upgrade is needed in order to use Dynamic Allocation and Auto-scaling working. I’ve decided to go straight for the most recent EMR release version, which as of February 2018 is EMR 5.11.1

To know more on EMR release versions, and see details, go HERE.

What changed on Spark?

Mainly, two things are different on Spark 2.2.1 in our scenario:

  • Spark stopped distributing Kinesis API’s because of licensing/legal issues. Jira HERE.
  • AWS deprecated a method we used on our streaming API to get the region based on the Kinesis endpoint. Javadoc HERE. Changes needed for SparkFeeding HERE.

To solve the first, we can use --packages org.apache.spark:spark-streaming-kinesis-asl_2.11:2.1.0 to add spark-streaming-kinesis-asl and its dependencies into the classpath automatically, rather than building the assembly jar by ourselves. However, for SBT to don’t fail on artifact building, it is required to use the %provided keyword for that library on `build.sbt`.

To solve the second issue, the `getRegion` method o the same class can be used. That one takes a string with the name of the region, which can be easily set on the spark config with the following CloudFormation property:

«spark.search.region»: {
«Ref»: «AWS::Region»
},

Getting YARN Logs on EMR

All this investigation was possible, by taking a look at the YARN error logs. If there is a need to take a look at the reason of a failed Spark Application.

Having trouble troubleshooting a failed Spark Application like Spark_Feeding? EMR will gather stderr and controller logs and upload them to S3, but most of the time the provided information will not be enough. So, here’s how you can get it:

  1. Go to the EMR cluster’s page in AWS
  2. In the summary page, note the Master’s public DNS.
  3. Fix the URL (i.e. ip-172-00-0-00.ec2.internal: … => 172.00.0.00: …)
    1. Tip: There is an add-on for Chrome that fixes it for you!
  4. SSH as hadoop to that address, providing the team’s SearchInfra_Search credentials file.
  5. You can now issue YARN commands to list applications and get the complete logging

Example commands

  • Get active applications details: yarn application --list
  • Get failed applications details: yarn application --list -appStates FAILED
  • Get an application full logs: yarn logs --applicationId <applicationID>

Firefox Quantum y el lenguaje Rust

Rust es un lenguaje de programación compilado, de propósito general y multiparadigma que está siendo desarrollado por Mozilla.​ Ha sido diseñado para ser «un lenguaje seguro, concurrente y práctico».​​

El Logo de Rust

El nuevo Firefox Quantum incluye Stylo, un motor de CSS escrito en Rust, que aprovecha al máximo la «Concurrencia Libre de Miedo» de Rust para acelerar el estilizado de la página. Es el primer componente principal de Servo que se integra con Firefox y representa un hito importante para Servo, Firefox y Rust. Reemplaza aproximadamente 160,000 líneas de C ++ con 85,000 líneas de Rust.

Firefox Quantum

 

Se puede leer mas al respecto de Rust y su «Fearless Concurrency» en el blog de Rust: https://blog.rust-lang.org/2017/11/14/Fearless-Concurrency-In-Firefox-Quantum.html

 

Mi franela del HacktoberFest ya llegó!

Cada octubre, Digital Ocean y GitHub envían camisetas gratuitas del Hacktoberfest a miles de personas en el mundo.

El diseño de este año.

El año pasado me quedo corto por un PR (Pull-Request), pero este año califique, y debo decir que la camiseta se ve GENIAL! Obtener una es sencillo: Debes realizar 4 solicitudes de extracción (PR, Pull-Requests) a proyectos Open Source en GitHub. Aquí lo resumo en 3 pasos:

  1. Registrarse en el Hacktoberfest:
    Necesitas una cuenta de GitHub. Si todavía no tienes uno, puedes crear uno gratis en solo unos minutos. Luego hay que ir a la página de registro del Hacktoberfest y seguir las instrucciones. Sólo toma un minuto.
  2. Realizar 4 PR’s:
    Se deben realizar cuatro Pull Requests a proyectos de Software libre. Pueden ser cosas tan sencillas como el archivo Readme.md de un repositorio publico cualquiera. Si no conoces ninguno, dejo a la orden mis repositorios. Aquí hay una guía más detallada para contribuir al código abierto, si te sientes aventurero.
  3. Verificar si has calificado:
    Una vez que se haya registrado, puede verificar su progreso en el sitio web de Hacktoberfest.

Python, primero en el ranking de la IEEE

Fuente: IEEE

Python ha continuado su trayectoria ascendente desde el año pasado y saltó dos lugares a la ranura N º 1, aunque los cuatro primeros (Python, C, Java y C++) – siguen estando muy cerca de popularidad.

Sin embargo, por segundo año consecutivo, no se han introducido nuevos idiomas en el ranking:

Parece que hemos entrado en un período de consolidación en la codificación como programadores digerir las herramientas creadas para atender a la explosión de la nube, móviles y grandes aplicaciones de datos.

Fuente: https://spectrum.ieee.org/static/interactive-the-top-programming-languages-2017