Overview
Lacework’s workload security provides visibility into all processes and applications within an organization’s cloud environments such as workload/runtime analysis, and automated anomaly and threat detection.
After you install the Lacework agent, Lacework scans hosts and streams select metadata to the Lacework data warehouse to build a baseline of normal behavior, which is updated hourly. From this, Lacework can provide detailed in-context alerts for anomalous behavior by comparing each hour to the previous one. Anomaly detection uses machine learning to determine, for example, if a machine sends data to an unknown IP, or if a user logs in from an IP that has not been seen before.
Lacework offers two methods for deployment into AWS ECS Fargate: container image layer-based and sidecar-based using a volume map approach. For both deployment methods, the Lacework agent runs inside the application container.
For container image layer-based deployments, when the agent layer is built, this starts the application container, which starts the Lacework agent in the context of the container as a daemon.
For sidecar-based deployments, the Lacework agent sidecar container exports a storage volume that is mapped by the application container. By mounting the volume, the agent can run in the application container namespace.
Prerequisites
- AWS Fargate platform versions 1.3 or 1.4
- OS distributions
- See public documentation for supported OS distributions
- Lacework agent 3.2 or later
Available from https://github.com/lacework/lacework-agent-releases
Deployment Methods
Lacework for AWS Fargate environment security monitoring supports two deployment methods using AWS Fargate:
- Container Image Layer-based Deployment
- Sidecar-based Deployment using a volume map approach.
Access Tokens
In an environment with Fargate and non-Fargate Lacework deployments, Lacework recommends using different access tokens for Fargate and non-Fargate deployments. This can make deployments easier to manage.
Container Image Layer-based Deployment
Container image layer-based deployment is recommended for scenarios where Lacework agents can be directly embedded into the container images being built. The following steps are based on AWS ECR deployment, but you can use similar steps for other registry types.
Predeployment for AWS ECR
Before attempting to publish a container to your registry, ensure you have the following items and information:
- Subnet ID
- Security group with correct permissions
- AWS CloudWatch log group
- Configure the ECS task execution role in IAM
- IAM policy
Attach the existing AmazonECSTaskExecutionRolePolicy to your ECS task execution role. - Trust relationship
- IAM policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Agent Upgrades
For Docker versions 17.05 and later with multi-stage builds, the latest agent release is used anytime you revision to a newer version of the application container. The runtime upgrade for the agent is available as well.
For Docker versions pre-17.05, when the agent is running, it always checks for the current version and downloads the latest. The version and git hash are available on the public github agent release page and also encoded in the S3 path.
Deployment Steps
In your chosen container registry, you create and publish the application container image (docker build, docker tag, docker push). Then define an AWS Fargate cluster and one or more task definitions associated with that cluster.
- Log in to an ECR registry.
$ aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin YourAWSId.dkr.ecr.us-west-2.amazonaws.com
- Build the Docker image. For an example Dockerfile that builds a container with the Lacework agent, see Example Dockerfile.
$ docker build --force-rm=true -t "lacework/rsfargate" YOUR_DIRECTORY_PATH_TO_DOWNLOADED_FILES
- Tag the Docker image on ECR.
$ docker tag lacework/rsfargate:latest YourAWSId.dkr.ecr.us-west-2.amazonaws.com/lacework/customerdemo:latest
- Push the Docker image to ECR.
$ docker push YourAWSId.dkr.ecr.us-west-2.amazonaws.com/lacework/customerdemo:latest
- Create a cluster. You only need to do this once.
$ aws ecs create-cluster --cluster-name cli-rsfargate-cluster
- Register a task definition, which describes the containerized workloads that you want to run.
For an example task definition, see Example Task JSON Definition.
$ aws ecs register-task-definition --cli-input-json file://YourPathToTaskJSonFile/fargate-task.json
Diagnostics
List Task Definitions
$ aws ecs list-task-definitions
Create a Service
$ aws ecs create-service --cluster cli-rsfargate-cluster --service-name cli-rsfargate-service --task-definition cli-run-task-definition:1 --desired-count 1 --launch-type "FARGATE" --network-configuration "awsvpcConfiguration={subnets=[subnet-abcd1234],securityGroups=[sg-abcd1234]}"
List Services
$ aws ecs list-services --cluster cli-rsfargate-cluster
Describe Running Services
$ aws ecs describe-services --cluster cli-rsfargate-cluster --services cli-rsfargate-service
Examples
Example Dockerfile (Using multi-stage builds - Docker versions 17.05 and later)
NOTE: This example uses crawl as a reference application to generate network connections and load.
FROM lacework/datacollector:latest-sidecar as laceworkagent
FROM ubuntu:18.04
COPY --from=laceworkagent /var/lib/lacework-backup /var/lib/lacework-backup
RUN apt-get update && \ apt-get install dnsutils curl jq -y && \
apt-get clean
COPY crawl /usr/bin/crawl
COPY run.sh /usr/bin/run.sh
RUN chmod +x /usr/bin/crawl && \
chmod +x /usr/bin/run.sh && \
mkdir -p /var/lib/lacework/config && \
echo '{ "tokens": {"accesstoken":"YourAccessToken"}}' > /var/lib/lacework/config/config.json
CMD [“sh”, “-c”, “/var/lib/lacework-backup/lacework-sidecar.sh && /usr/bin/run.sh”]
Example Service/Application Run Script
NOTE: This example uses crawl as a reference application to generate network connections and load.
#!/bin/sh
ulimit -n 4096
sleep 30s
loops=1000
runs=300
for count in `seq ${runs}`;
do
echo "Iter $count"
crawl www.abcnews.com ${loops} & >> /dev/null
wait
sleep 1m
done
echo "Finished..."
Example Dockerfile (Docker versions pre-17.05)
NOTE: This example uses crawl as a reference application to generate network connections and load.
FROM ubuntu:18.04
RUN apt-get update && \
apt-get install dnsutils curl jq -y && \
apt-get clean
ENV CommitHash 3.1.36_2020-08-28_master_c20c2bbeb473cc019611a473de5cc9694bb83757
ENV DcVersion 3.1.36
COPY crawl /usr/bin/crawl
COPY run.sh /usr/bin/run.sh
RUN chmod +x /usr/bin/crawl && \
chmod +x /usr/bin/run.sh && \
mkdir -p /var/lib/lacework/${DcVersion} && \
mkdir -p /var/lib/lacework/config && \
mkdir -p /var/log/lacework && \
curl -sSL https://s3-us-west-2.amazonaws.com/www.lacework.net/download/${CommitHash}/datacollector.gz | gunzip > /var/lib/lacework/${DcVersion}/datacollector && \
chown root:root /var/lib/lacework/${DcVersion}/datacollector && \
chmod +x /var/lib/lacework/${DcVersion}/datacollector && \
echo '{ "tokens": {"accesstoken":"YourAccessToken"}}' > /var/lib/lacework/config/config.json && \
ln -s /var/lib/lacework/${DcVersion}/datacollector
CMD /usr/bin/run.sh
Example Service/Application Run Script
NOTE: This example uses crawl as a reference application to generate network connections and load.
#!/bin/sh
ulimit -n 4096
/var/lib/lacework/datacollector &
sleep 30s
loops=1000
runs=300
for count in `seq ${runs}`;
do
echo "Iter $count"
crawl www.abcnews.com ${loops} & >> /dev/null
wait
sleep 1m
done
echo "Finished..."
Example Task JSON Definition
{
"executionRoleArn": "arn:aws:iam::775731770260:role/ecsTaskExecutionRole",
"containerDefinitions": [
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/cli-run-task-definition",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
},
"cpu": 0,
"environment": [
{
"name": "LaceworkAccessToken",
"value": "YourAccessToken"
}
],
"image": "YourAWSId.dkr.ecr.us-west-2.amazonaws.com/lacework/customerdemo:latest",
"essential": true,
"name": "cli-rsfargate"
}
],
"memory": "1GB",
"family": "cli-run-task-definition",
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc",
"cpu": "512"
}
Sidecar-based Deployment
You can also deploy Lacework for AWS Fargate using a sidecar approach, which allows you to monitor AWS Fargate containers without changing underlying container images. This deployment method leverages AWS Fargate’s VolumesFrom capabilities to map a predefined Lacework container image to be run alongside your production workloads.
Predeployment
Before attempting to publish a container to ECR, ensure you have the following items and information:
Configure the ECS task execution role to include AmazonECSTaskExecutionRolePolicy.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "*" } ] }If it is not already configured, you may need to include some of the ECS access policies as well, such as AmazonECS_FullAccess.
Agent Upgrades
When running, the agent always checks for the current version and downloads the latest. The sidecar container always points to the latest in hub.docker.com, for example, lacework/datacollector:latest-sidecar.
Task Definition
A task definition describes the containerized workloads that you want to run.
Create a new task definition.
Add task details.
Set the task execution role.
Set memory and CPU requirements for the application.
Sidecar Container Definition
- Click Add container to add the agent sidecar container.
In the Environment section, deselect the Essential checkbox. This is not an essential container.
You do not need to change any fields in the remaining sections.Click Add to finish adding this sidecar container.
Container Definitions now lists the new container.
Application Container Definition
Click Add container to add the application container.
Ensure the application container has a different name than the Lacework agent sidecar (datacollector) container.
- In the Environment section, ensure Essential is selected.
Before continuing, you must identify whether the container uses ENTRYPOINT and/or CMD. Ideally, you could examine the Dockerfile used to build the container to identify ENTRYPOINT and/or CMD usage in building the container. Alternatively, you must use tools to inspect the Docker container image to identify ENTRYPOINT and/or CMD usage.
For example, using the Docker Hub interface, you can inspect the container image layers to identify the usage.
The following figure illustrates an Ubuntu container using the CMD directive.
The following figure illustrates an Nginx container using the ENTRYPOINT and CMD directive.
- After identifying whether the container uses ENTRYPOINT and/or CMD, place an appropriate definition in the ENTRYPOINT or CMD container
definition section of ENVIRONMENT. Depending on the application container, in order to sidecar the Lacework agent in the application, place the
following definition.
sh, -c, /var/lib/lacework-backup/lacework-sidecar.sh && APPLICATION_INIT
NOTE: By definition, ENTRYPOINT + CMD = default container command with arguments. It is possible that the override of ENTRYPOINT + CMD may not provide the desired outcome. If the application container uses ENTRYPOINT, it is recommended to use ENTRYPOINT override with the Lacework sidecar because it takes precedence over CMD override.
The following figure illustrates an example that overrides the application container’s CMD section while using /usr/bin/runapp.sh as the startup wrapper for the application container.
- In the Startup Dependency Ordering section, ensure the application container has the dependency on the Lacework agent sidecar container so
that the application container only starts after the sidecar container successfully starts.
In the Storage and Logging section, ensure the application container imports the volume that is exported by the Lacework agent sidecar container. This makes the volume that contains the Lacework agent executable available in the application container.
For Volumes from Source container, add the sidecar container name. Also select the Read only checkbox.
- Click Add to finish this application container.
Container Definitions now lists both the sidecar and application containers.
- Click Create to create the task.
JSON Task Definition
The sidecar container is available in hub.docker.com as lacework/datacollector:latest-sidecar.
{
"requiresCompatibilities": [
"FARGATE"
],
"inferenceAccelerators": [],
"containerDefinitions": [
{
"name": "datacollector-sidecar",
"image": "YourAWSId.dkr.ecr.us-west-2.amazonaws.com/datacollector:latest-sidecar",
"resourceRequirements": null,
"essential": false,
"portMappings": [],
"environment": null,
"secrets": null,
"mountPoints": null,
"volumesFrom": null,
"hostname": null,
"user": null,
"workingDirectory": null,
"extraHosts": null,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/TestSideCarTaskDef",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
},
"ulimits": null,
"dockerLabels": null,
"dependsOn": null,
"repositoryCredentials": {
"credentialsParameter": ""
}
},
{
"name": "app-container",
"image": "YourAWSId.dkr.ecr.us-west-2.amazonaws.com/datacollector-sidecar-app:latest",
"resourceRequirements": null,
"essential": true,
"portMappings": [],
"command": [
"sh",
"-c",
"/var/lib/lacework-backup/lacework-sidecar.sh && /usr/bin/runapp.sh"
],
"environment": [
{
"name": "LaceworkAccessToken",
"value": "YourAccessToken"
}
],
"secrets": null,
"mountPoints": null,
"volumesFrom": [
{
"sourceContainer": "datacollector-sidecar",
"readOnly": true
}
],
"hostname": null,
"user": null,
"workingDirectory": null,
"extraHosts": null,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/TestSideCarTaskDef",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
},
"ulimits": null,
"dockerLabels": {
"Monitoring": "Lacework"
},
"dependsOn": [
{
"containerName": "datacollector-sidecar",
"condition": "SUCCESS"
}
],
"repositoryCredentials": {
"credentialsParameter": ""
}
}
],
"volumes": [],
"networkMode": "awsvpc",
"memory": "1024",
"cpu": "512",
"executionRoleArn": "arn:aws:iam::YourAWSId:role/ecsTaskExecutionRoleTest",
"family": "TestSideCarTaskDef",
"taskRoleArn": "arn:aws:iam::YourAWSId:role/ecsTaskExecutionRoleTest",
"tags": [
{
"key": "AppContainerSecurity",
"value": "Lacework"
}
]
}
Full JSON Task Definition with IAM Role
{
"ipcMode": null,
"executionRoleArn": "arn:aws:iam::YourAWSId:role/ecsTaskExecutionRoleTest",
"containerDefinitions": [
{
"dnsSearchDomains": null,
"environmentFiles": null,
"logConfiguration": {
"logDriver": "awslogs",
"secretOptions": null,
"options": {
"awslogs-group": "/ecs/TestSideCarTaskDef",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
},
"entryPoint": null,
"portMappings": [],
"command": null,
"linuxParameters": null,
"cpu": 0,
"environment": [],
"resourceRequirements": null,
"ulimits": null,
"dnsServers": null,
"mountPoints": [],
"workingDirectory": null,
"secrets": null,
"dockerSecurityOptions": null,
"memory": null,
"memoryReservation": null,
"volumesFrom": [],
"stopTimeout": null,
"image": "YourAWSId.dkr.ecr.us-west-2.amazonaws.com/datacollector:latest-sidecar",
"startTimeout": null,
"firelensConfiguration": null,
"dependsOn": null,
"disableNetworking": null,
"interactive": null,
"healthCheck": null,
"essential": false,
"links": null,
"hostname": null,
"extraHosts": null,
"pseudoTerminal": null,
"user": null,
"readonlyRootFilesystem": null,
"dockerLabels": null,
"systemControls": null,
"privileged": null,
"name": "datacollector-sidecar"
},
{
"dnsSearchDomains": null,
"environmentFiles": null,
"logConfiguration": {
"logDriver": "awslogs",
"secretOptions": null,
"options": {
"awslogs-group": "/ecs/TestSideCarTaskDef",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
},
"entryPoint": null,
"portMappings": [],
"command": [
"sh",
"-c",
"/var/lib/lacework-backup/lacework-sidecar.sh && /usr/bin/runapp.sh"
],
"linuxParameters": null,
"cpu": 0,
"environment": [
{
"name": "LaceworkAccessToken",
"value": "YourAccessToken"
}
],
"resourceRequirements": null,
"ulimits": null,
"dnsServers": null,
"mountPoints": [],
"workingDirectory": null,
"secrets": null,
"dockerSecurityOptions": null,
"memory": null,
"memoryReservation": null,
"volumesFrom": [
{
"sourceContainer": "datacollector-sidecar",
"readOnly": true
}
],
"stopTimeout": null,
"image": "YourAWSId.dkr.ecr.us-west-2.amazonaws.com/datacollector-sidecar-app:latest",
"startTimeout": null,
"firelensConfiguration": null,
"dependsOn": [
{
"containerName": "datacollector-sidecar",
"condition": "SUCCESS"
}
],
"disableNetworking": null,
"interactive": null,
"healthCheck": null,
"essential": true,
"links": null,
"hostname": null,
"extraHosts": null,
"pseudoTerminal": null,
"user": null,
"readonlyRootFilesystem": null,
"dockerLabels": null,
"systemControls": null,
"privileged": null,
"name": "app-container"
}
],
"placementConstraints": [],
"memory": "1024",
"taskRoleArn": "arn:aws:iam::YourAWSId:role/ecsTaskExecutionRoleTest",
"compatibilities": [
"EC2",
"FARGATE"
],
"taskDefinitionArn": "arn:aws:ecs:us-west-2:YourAWSId:task-definition/TestSideCarTaskDef:1",
"family": "TestSideCarTaskDef",
"requiresAttributes": [
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.execution-role-awslogs"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.task-iam-role"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.container-ordering"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.execution-role-ecr-pull"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.task-eni"
}
],
"pidMode": null,
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc",
"cpu": "512",
"revision": 1,
"status": "ACTIVE",
"inferenceAccelerators": null,
"proxyConfiguration": null,
"volumes": []
}
Run Task
- Select Fargate as the Launch type.
- Select your Platform version.
Fargate Information in the Lacework Console
Fargate Task Information
To view Fargate task information, navigate to Monitor > Agents then the Agent Monitor table. Note that the displayed hostname is not an actual host but rather an attribute (taskarn_containerID). Expand the hostname to display its tags, which contain Fargate task information, additional metadata collected by the agent.
This tag provides the Fargate engine version: VmInstanceType:AWS_ECSVxFARGATE
Where x can be 3 or 4 (1.3 or 1.4 engine respectively).
The following tags are prepended by net.lacework.aws.fargate.
| Tag | Description |
|---|---|
| cluster | The cluster where the task was started |
| family | The task definition family |
| pullstartedat | The time the container pull started |
| pullstoppedat | The time the container pull stopped. The diff from pullstartedat represents how long the pull took to start. |
| revision | The task revision |
| taskarn | The full task ARN |
You can also view the above tag information at Host > Applications in the List of Active Containers table. Make the Machine Tags column visible; it is hidden by default.
Fargate Container Information
You can find container information at Host > Applications in the List of Active Container table. The container ID is available, but AWS currently does not expose underlying infrastructure. To view tags, make the Machine Tags column visible; it is hidden by default.
The containers listed in Container Image Information are application containers. If you used the sidecar deployment method, the sidecar container itself is not displayed in this table because it is not running and has no runtime or cost associated with it. The Lacework application, however, runs as a process inside your application container, so it is visible in the Applications Information table (search for datacollector) because there is no running container for the Lacework agent.
Additional Notes
This section provides additional clarification about Lacework behavior with Fargate deployments.
FIM
By default Lacework does not perform file integrity checks within containers. This means that for a Fargate-only environment, there is no information under the Host > Files (FIM) menu. In an environment containing Fargate and non-Fargate deployments, the expected information displays. The checks are not performed by default because they are expensive for CPU and memory.
Event Triage
Lacework has the following methods for accessing Lacework events:
- Alert channels such as Email, Jira, Slack
- Lacework Console: Monitor > Events
Alert Channels
The following figure illustrates an email alert channel with 2 medium alerts.
Click View Details → to load the Event Details page in your default browser.
Lacework Console: Monitor > Events
Click Monitor > Events and set the time frame for your inspection. 
A summary of the currently selected event is available in the Event Summary in the upper right quadrant of the page.
This page presents the following major launch points:
- Event Summary—You can review high level event information. In the above figure, the description contains a destination host, a destination port, an application, and a source host. Each of these links provides a resource-specific view. For example, if you click the destination host, you can view all activity for that destination host seen by the Lacework Platform.
- Timeline > Selected Event > Details—This opens the Event Details page. The Event Details page is the same page that opens when you click the View Details link from an alert channel.
Triage Example
The following exercise focuses on the Event Details, which is illustrated in the following figure.
On this page you can view the 5W structure for Lacework events.
Key Data Points to Triage a Fargate Event
WHAT
Expand the details for WHAT.
Header Row
| Key | Value Description | Notes |
|---|---|---|
| APPLICATION | What is running | None |
| MACHINE | The Fargate ARN:task for the job | Machine-centric view |
| CONTAINER | The container the task is running | None |
Application Row
| Key | Value Description | Notes |
|---|---|---|
| APPLICATION | What is running | None |
| EARLIEST KNOWN TIME | First time an application was seen | Useful for lining up incident responses with timelines in non-Lacework services. |
Container
| Key | Value Description | Notes |
|---|---|---|
| FIRST SEEN TIME | First time a container was seen | Useful for lining up incident responses with timelines in non-Lacework services. Useful for spot checks of your expected container lifecycle versus what is running. |
WHERE Expand the details for WHERE. There is a WHERE section if the event is network related.
| Key | Value Description | Notes |
|---|---|---|
| HOSTNAME | Pivot to inspect the occurrence of the host activity within your environment | None |
| PORT LIST | Ports activity was seen over | Context. Do you expect this service to connect over these ports? Are there any ports that are interesting, such as non standard ports for servers. |
| IN/OUT BYTES | Data transmitted/received | Inferring risk based on payload size. |
Investigation
High level situational context that relates to the application deployment and your knowledge of its expected behavior.
Was the application involved in the event from Packaged software? In this case, the event is for nginx and tomcat. If you install tomcat / nginx using the package manager, this is marked as yes. If you install tomcat / nginx using the package manager and this is marked no, investigate if there was a change to how you build your container. If there is no change to your build process, investigate how a non packaged version was installed.
Has the Hash(SHA256) of the application been involved in an event change? Was the application involved altered / replaced. If so, check your build and upgrade process. It is typical that containers will not see application updates. A change here warrants continued investigations to determine the cause of change.
Has the application involved in the event been running as Root? Indicator of the scope of privilege the application has. If running as root is not part of your build processes, check for an alteration to your build processes and privilege escalation.
Has the application transferred more than the median amount of data compared to the last day?
Consider the amount of data transferred versus what is expected by the application.
Related Events Timeline The timeline is useful for building context for events around the Event Details you are currently investigating.
Summary
Using the above data, you should be able to create the following triage process.
Determine the scope of the event:
- Containers
- Applications
- Hosts
- arn:task
Drill down into the event relationships to get context across your infrastructure secured by the Lacework platform.
Determine what is expected as a part of your build processes and what is unexpected.
From this path, continue to narrow down if there was a change to build processes or a net new build process or infrastructure event/service. For these scenarios, focus on updating application, build, and network configurations.
For scenarios where there is a risk of compromise, focus on the timeline that led to the potential compromise, investigate the entities involved, and create a structured story that enables rapid IR and processes updates.