Brian Klaas

Slides from My ColdFusion Summit Presentation on Creating Complex Serverless Workflows with CFML at the Center

2022-10-12T08:39:00-04:00

This year marked the tenth anniversary of the Adobe ColdFusion Summit. It was great to get back together with so many familiar faces (and friends!) in person. I’ve attended all of the ColdFusion Summits and, I think, have spoken at all of them as well. I spoke at Summit this year about my favorite topic: ColdFusion and Amazon Web Services.

“Serverless Beyond Simple: Creating Complex Workflows with ColdFusion at the Center” is an architectural overview of what it takes to build resilient, complex serverless workflows in AWS that are controlled and managed by ColdFusion. The presentation is a distillation of many years of experience in building ColdFusion-based serverless workflows. Even if you’re just getting started with serverless technologies, there’s useful guidance in there for you. The presentation covers Simple Queue Service (SQS) vs. Simple Notification Service (SNS), the power of orchestration in AWS Step Functions, and managing the state of workflows in both locally and in DynamoDB. I’d be remiss if I didn’t also say (as I did in the presentation) that the architectural patterns also apply to building workflows in Google Cloud and Azure.

A PDF of the slides from the presentation is available. None of this year’s ColdFusion Summit sessions were recorded, so there is no video for me to share with you.

If you have any questions about this content in this presentation, feel free to reach out to me on the Twitter!

Slides and Recording from My ColdFusion Developer Week Presentation on Advanced Features of S3

2022-08-16T12:18:00-04:00

I’m grateful that I’ve spoken at Adobe’s ColdFusion Developer Week over the years. This is a great opportunity for those interested in Adobe ColdFusion or the CFML language to get a wide variety of free training about ColdFusion and CFML. I spoke at CF Dev Week again this year about my favorite topic: ColdFusion and Amazon Web Services.

Although I spoke about Simple Storage Service (S3) at CF Dev Week in years past, this presentation looked at how to take advantage of more advanced features of S3 using the native features of Adobe ColdFusion 2021. All of the code samples in the presentation work in Adobe ColdFusion 2021. If you happen to run ColdFusion 2018, 2016, or even ColdFusion 11, my AWS Playbox repo has code which shows you how to do the very same things using the very powerful AWS Java SDK.

Below are the links to the two resources from the presentation: the slides and the recording of the presentation on YouTube:

A PDF of the slides from the presentation
Presentation recording on YouTube

If you have any questions about this content in this presentation, feel free to reach out to me on the Twitter!

Resources from Scalable, Responsive Apps and Services with Queues and Pub/Sub Mechanisms

2021-12-27T08:18:00-05:00

In spite of the pandemic, the Adobe ColdFusion team held not one but two virtual conferences this calendar year about all things ColdFusion. In addition to helping organize the speakers, I also spoke at both versions of the conference. As usual, I spoke about leveraging the power of cloud-based services to improve on the scalability, reliability, and performance of your ColdFusion applications. This year’s topic was “Scalable, Responsive Apps and Services with Queues and Pub/Sub Mechanisms,” which focused on Amazon Web Services’ Simple Queue Service and Simple Notification Service, both of which are easy to use from within ColdFusion.

Below are the resources from the presentation: a PDF copy of the slides, and a link to my AWS Playbox repo on GitHub. The AWS Playbox repo has examples of using both SQS and SNS via the AWS Java SDK. This is the route you have to go if you’re using any version of ColdFusion prior to ColdFusion 2021.

The ColdFusion team will post videos from all sessions at the conference on the ColdFusion Community portal. Until they do that, though, you can view the recordings on the conference site. The always-helpful Charlie Arehart has a post on the community portal that walks you through finding the recordings.

A PDF of the slides from the presentation
AWS Playbox Repo on GitHub

If you have any questions about this content in this presentation, feel free to reach out to me on the Twitter!

Resources from My ColdFusion Summit 2020 Presentation on cflambda

2020-11-17T08:18:00-05:00

This week, I’m speaking at the ColdFusion Summit 2020 about “Building Serverless ColdFusion Applications with cflambda.” cflambda is a new release from the Adobe ColdFusion team that’s part of ColdFusion 2021 and which lets you write AWS Lambda functions in ColdFusion. There are a number of resources referenced in the presentation that I’d like to share in this post:

A PDF of the slides from the presentation
The cflambda function code from the presentation
Paul Kukiel’s sample API Gateway Lambda repo
AWS SAM (Serverless Application Model)

More examples of calling Lambda functions — and other AWS services — from your ColdFusion applications can be found in my AWS Playbox repo on GitHub.

Below are the recordings of two related presentations that I mentioned during this talk:

If you have any questions about this content in this presentation, feel free to reach out to me on the Twitter!

Slides and Recording from my CFMeetup Presentation: I Didn’t Know S3 Could Do That!

2020-08-10T09:08:00-04:00

Thanks to the amazing Charlie Arehart for hosting me on last week’s CFMeetup. My presentation — “I Didn’t Know S3 Could Do That!” — covered a lot of the same material that is in my recent series on using S3 in CFML. There were a lot of really good questions, and I very much enjoyed my time delivering the presentation.

You can download the slides from the presentation, or watch the whole presentation on YouTube.

Beyond the Basics of Using AWS S3 in CFML: Editing Tags and Using S3 Object Metadata

2020-08-03T09:55:00-04:00

In the last post in this series, we looked at how to add tags to an object uploaded to S3 via the AWS Java SDK. What if you need to change those tags? Are tags the only way to add metadata to objects in S3? Let’s file out.

Updating Object Tags in S3

It’s possible to update the tags assigned to an object in S3. This is particularly important if you already have files in S3 without tags. To do this, you’ll tell the S3 client in your application to run a setObjectTagging() request:

tag = CreateObject('java', 'com.amazonaws.services.s3.model.Tag').init(tagKey, tagValue);
fileTagging = CreateObject('java', 'com.amazonaws.services.s3.model.ObjectTagging').init([tag]);
// Remember: "key name" is the path to the file (incuding the file name) in the bucket
objectTaggingRequest = CreateObject('java', 'com.amazonaws.services.s3.model.SetObjectTaggingRequest')
    .init(bucketName, keyName, fileTagging);
s3.setObjectTagging(objectTaggingRequest);

It’s important to note that when you run a setObjectTagging() request, it replaces the existing tags on the object. There’s no way to just “update” the tags on an object.

Tags vs. Metadata

When you put a file in S3, S3 automatically adds quite a bit of metadata, including:

Content-Length
Content-Type
Last-Modified
x-amz-server-side-encryption
x-amz-storage-class

and more. If some of these look like regular, plain http response headers, that’s because they are.

If you want to add your own custom metadata to an object in S3, you can add metadata instead of tags. Tags are not the same thing as object metadata in S3. Metadata applies only to that object in S3 and cannot be searched on, as you can search with tags.

To set an object’s metadata, you would set the objectMetadata object on your putFileRequest, as follows:

objectMetadata = CreateObject('java', 'com.amazonaws.services.s3.model.ObjectMetadata').init();
objectMetadata.setUserMetadata(plainOldCFStructOfMetaData);
putFileRequest.setObjectMetadata(objectMetadata);

You can read the metadata for an object in S3 when you run a getObject() or GetObjectMetadataRequest() request. The metadata is also sent in the response headers of every plain http request to an object in S3 (ie; https://somebucket.s3.amazonaws.com/somePhoto.jpg).

It’s important to note that you cannot edit or change the metadata for an object once that object is created. If you need to do this, you have copy the object into the same place (effectively overwriting the file) with a CopyObjectRequest object with the new metadata. It’s not convenient, but if you have metadata that you want to come down with every plain http request to an object in S3, metadata is the only way to go. Tags are not included in plain http requests to objects in S3.

That’s it for object tags and metadata in S3. As always, if you have any questions about this, feel free to message me on the Twitter.

Beyond the Basics of Using AWS S3 in CFML: The Why and How of Adding Tags to Your Files

2020-07-27T08:51:00-04:00

As anyone who has worked in any of the major cloud service providers will tell you, there will be ever-increasing sprawl the more you use a cloud provider: more files, more servers, more databases, more accounts, more services, more everything. This is especially the case with object storage like S3. You can easily end up with tens or hundreds of thousands of files in a month. If you were to look at any one of the files in a bucket, could you say who put that file there? Could you look at all the files and be able to charge each file owner for the storage cost? Not without the power of tags.

Tags in AWS

Tags are metadata-like key-value pairs you can apply to nearly anything in AWS. You determine the key and you determine the value and then apply the tags to a resource in AWS. You can then use those tags when you search for resources (including files in S3) inside AWS. You can even enable a special kind of tag — a cost-allocation tag — that will be used to break down individual “ownership” costs of each service on your monthly AWS bill. Most people use tags to determine things like the:

Department which owns the resource (“department” : “marketing”)
Environment in which the resource runs (“environment” : “dev”)
Application to which the resource belongs (“application” : “order fulfillment”)
Security level of the resource (“compliance” : “hippa”)

Tags can represent anything, so it’s important that you:

a. start tagging your resources as soon as possible
b. have a strategy for coming up with tags

AWS sees how hundreds of thousands of customers tag their resources and have come up with a really useful guide to tagging strategies. It’s worth a read when you’re coming up with your own plan for tagging.

Adding Tags to Files in S3

Adding tags to files in S3 is very much like adding tags to any resource in AWS. You create tags, and then add them to a request to create or modify a resource in AWS. In the case of S3, you create your tags, add them to an ObjectTagging object, and then add that ObjectTagging object to the putObject request that actually uploads your file to S3.

s3 = application.awsServiceFactory.createServiceObject('s3');
javaFileObject = CreateObject('java', 'java.io.File').init(fileLocation);
putFileRequest = CreateObject('java', 'com.amazonaws.services.s3.model.PutObjectRequest')
     .init(bucketName, fileName, javaFileObject);

// Here's where we add our tag
tag = CreateObject('java', 'com.amazonaws.services.s3.model.Tag').init(tagKey, tagValue);
fileTagging = CreateObject('java', 'com.amazonaws.services.s3.model.ObjectTagging').init([tag]);
putFileRequest.setTagging(fileTagging);

s3.putObject(putFileRequest);

The code above should be fairly self-explanatory, but here’s a quick rundown. First, you create a putFileRequest using a standard Java File object. (Read more about uploading files to S3 via the AWS Java SDK.) Next, you create a Tag object, assigning it the key and value with which you want to tag the file. Next, the tag(s) are added to an ObjectTagging object, which is a standard object throughout the AWS Java SDK. Finally, you set the tagging on the putFileRequest object using the ObjectTagging object. When the file is actually uploaded to S3, the tags are added to the file.

It’s really straightforward to add tags to the objects you put in S3, so start doing it! In the next post in this series, we’ll look at editing object tags, and the differences between tags and object metadata in S3.

Beyond the Basics of Using AWS S3 in CFML: File Versioning in S3 Buckets

2020-07-13T10:01:00-04:00

When editing files of any kind, users of modern applications expect to have access to previous versions of the file, and undo work that they don’t like or made in error. Implementing a versioning system isn’t easy, especially when file systems are generally set up to only allow one “copy” of any given file. It’s often up to you, the developer, to handle the versioning and ensure that new versions of uploaded files don’t overwrite older ones.

Fortunately, it’s exactly this kind of undifferentiated heavy lifting at which AWS excels. Why should you worry about file versioning at the file system level, when you can utilize this as a cheap service? Versioning is built in to S3, and comes at no additional cost — except for all the copies of the files that you store.

Enabling File Versioning in CFML

Versioning is enabled on per-bucket basis. There’s no way to say “just version this file,” or even provide a path prefix that says “only affect these files in this path,” as you can do with lifecycle rules.

As always, the full code for this and all of my demos can be found in my AWSPlaybox GitHub repo.

Here’s how you enable versioning on a bucket in CFML:

s3 = application.awsServiceFactory.createServiceObject('s3');
bucketVersioningConfig = CreateObject('java', 'com.amazonaws.services.s3.model.BucketVersioningConfiguration')
    .withStatus('Enabled');
bucketVersioningConfigRequest = CreateObject('java', 'com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest')
    .init(bucketName, bucketVersioningConfig);
s3.setBucketVersioningConfiguration(bucketVersioningConfigRequest);

It’s quite simple: you create a bucket versioning config object, set the status to “enabled” and tell the S3 bucket to utilize that new configuration.

Once you have versioning enabled, S3 will store a new version every time you POST a new version of the same file (the same file path/”prefix”) in the same bucket. The response object returned from a PutFile() request to S3 using the AWS Java SDK will give you the version ID of that version of the file which you just uploaded. You’ll need to store that ID in your own application along with some kind of metadata about what that version of the file represents. S3 will not tell you the differences in the content between different versions of a file. It’s not Git. That’s not what it is designed to do. It simply shows all the versions of the file in the bucket. It’s up to you to keep track of what each version represents, or a summary of changes between versions.

Listing Versions of a File in CFML

It’s pretty straightforward to retrieve the versions of a file:

listVersionsRequest = CreateObject('java', 'com.amazonaws.services.s3.model.ListVersionsRequest')
    .withBucketName(bucketName)
    .withPrefix(pathToFileForVersioning)
    .withMaxResults(5);
versionsResult = s3.listVersions(listVersionsRequest);
summariesArray = versionsResult.getVersionSummaries();
summariesArray.each(function(objectSummary) {
    resultData &= "Key: " & objectSummary.getKey() & ", versionID: " & objectSummary.getVersionId();
});

The ListVersionsRequest object is what does the work. It has a number of methods for specifying the data you want returned, but the primary required property (beyond the bucket name) is set via the withPrefix() method. It’s here that you specify the path to the file in the bucket. You can think of a path to a file in a versioned bucket like a folder that contains all the versions of a file therein. The ListVersionsRequest simply shows you all the files in that “folder.” Using the withMaxResults() method limits you to the [n] most recent versions of that file.

Once you tell S3 to list the verisons for that prefix, it returns, within the request result object, an array of summaries about each version of the file. Again, it’s not going to tell you what’s different in the content between each version. You’ve got to track that on your own. You can loop through the summaries array to get the IDs of each version of the file (the “key”) that you will match to data in your own application about what each version represents.

Warning: Cost Increases Ahead!

It’s important to remember that while you get all of the versioning functionality in S3 for free, you pay for every version of every file stored. If a file is deleted out of a bucket that has versioning enabled, the file isn’t actually deleted. A deletion marker is placed on the file so that it can be un-deleted at a later time. If versioning is disabled on a bucket, all the old versions of each file are retained for safety’s sake.

All of these versions of files start to add up over time. It’s important to delete older versions of files. There is no way to limit the number of versions kept, or to set a maximum number of versions per file and automatically delete files older than that maximum.

You’re best off building tasks to check for old versions manually, or to use a lifecycle rule to have old versions automatically expire after a period of time. The BucketLifecycleConfiguration.Rule. setNoncurrentVersionExpirationInDays(int) method will help you do this transparently.

Versioning is a powerful tool and an easy addition to any S3 bucket. Just watch out for those unintented cost increases. As always, if you have any questions about this, feel free to message me on the Twitter.

Beyond the Basics of Using AWS S3 in CFML: Use Lifecycle Rules to Automatically Delete Files After a Period of Time

2020-06-22T13:08:00-04:00

In the last post in this series, we looked at Lifecycle Rules in S3, how they worked, and how you can use them to save money by automatically moving less-frequently used files to different (and cheaper) storage classes. There’s another powerful aspect to Lifecycle Rules that can save you money: automatically deleting files after a specified period of time.

Many of the files that are added to S3 buckets are never used after a brief, initial period of time. Many companies put all of their log files in S3. Are those log files valuable after data analysis is complete? Are the storage costs worth the price after 180 days? After 365? Are there regulatory or compliance reasons to not store user-generated data for a long period of time?

Whatever the reason, do you want to do the work of tracking every file in S3 and then running tasks to determine if the file is ready for deletion? Wouldn’t it be nice if all of that could be handled transparently for you? With S3 expiration actions, you don’t have to do this work. Expiration actions are a type of Lifecycle Rule that tells S3 to automatically delete a file in a bucket after a specified period of time.

Enabling Expiration Actions in CFML

The process of enabling expiration actions is the same as enabling transition actions. The same steps for setting up transition actions apply when setting up expiration actions. As always, the full code for this and all of my demos can be found in my AWSPlaybox GitHub repo.

If you haven’t reviewed the last post to see the steps required in setting up transition actions, please do so as I will not repeat them here.

The CFML code for enabling an expiration action is as follows:

bucketLifecycleConfig = CreateObject('java', 'com.amazonaws.services.s3.model.BucketLifecycleConfiguration').init();
rule = CreateObject('java', 'com.amazonaws.services.s3.model.BucketLifecycleConfiguration$Rule').init();
// The rule Id just helps you to identify the purpose of the rule in plain text
rule.setId("Delete objects after 90 days");
rule.setExpirationInDays(90);
// Rules must be set to ENABLED to actually function on the bucket
rule.setStatus(bucketLifecycleConfig.ENABLED);
bucketLifecycleConfig.setRules([rule]);
s3.setBucketLifecycleConfiguration(s3BucketName, bucketLifecycleConfig);

This code is a bit simpler than the code for transition actions, because we’re not enabling any transitions. We’re simply setting an expiration action that applies to every file in the bucket. In this case, any file in the bucket will be automatically deleted after 90 days.

You can combine transition actions and expiration actions in the same rule. You can create a rule that says “Move files from S3 standard to S3 One Zone Infrequent Access after 90 days, and delete all files after 180 days.” That’s exactly what the example code in the AWS Playbox app does.

As explained in the post on transition actions, you can only have one rule set that applies to all files in a bucket. You can have multiple filters based on object path prefixes or tags in a single bucket, and any combination of transition and expiration actions in those filters.

Lifecycle Rules are a powerful way to reduce your overall S3 costs. If you’re storing more than a few hundred gigabytes of data in S3, I encourage you to start using them. As always, if you have any questions about this, feel free to message me on the Twitter.

Beyond the Basics of Using AWS S3 in CFML: Use Lifecycle Rules to Move Files Into Different Storage Classes

2020-06-16T13:48:00-04:00

In my last post in this series, we looked at using different storage classes in S3 via CFML to save money on long-term storage costs. The use case for that blog post assumed you knew the utilization pattern for the files you uploaded. They would either be very busy files, files that were accessed occasionally, or files that were accessed rarely. What happens, though, when file acess moves over time from frequently access to only occasional access? Wouldn’t it be nice if S3 would automatically move those files from a more expensive storage class with the fastest response time to a storage class where response times were a few miliseconds longer, but cost half the price?

S3 provides lifecycle rules to do exactly that.

How Lifecycle Rules Work

You can only apply lifecycle rules to a whole bucket, but you can define filters in these rules so that only specific files in the bucket are affected by the lifecycle rule. There are two types of lifecycle rules that can be applied to a S3 bucket. There are transition actions, which we’ll look at now, and expiration actions, which we will look at in a future post.

Transition actions tell S3 to transition files from one storage class to another when certain criteria are met. For example, you can tell S3 to automatically move files to infrequent access, one zone storage 30 days after creation. You can tell S3 to automatically move files to Glacier deep archive after 365 days. This not only can save you money over time (do you really need the drafts of the 2014 annual report to stay in S3 standard storage?), but it also automates determining which files should be moved and when they should be moved.

Intelligent Tiering vs. Transition Actions

S3 does offer a storage class called “intelligent tiering.” This tier of storage looks at access patterns for every file in this storage class, and automatically moves files from S3 standard to S3 infrequent access if the file has not been accessed in the last 30 days. The advantage to intelligent tiering is that if a file remains frequently accessed after 30 days, it will stay in the S3 standard class. Transition actions ignore individual file access patterns. Transition actions are apply to all files in the bucket, regardless of individual file access patterns. If your lifecycle rule says to transition all files in the bucket from S3 standard to S3 infrequent access after 30 days, that’s going to happen at the 30 day mark even if the file is still frequently accessed.

Intelligent tiering does have an overhead charge of $0.0025 per 1,000 objects monitored. This can add up over time, and may not be the best choice if your general access patterns for files in a bucket are well known. Intelligent tiering only manages files over 128KB in size, so this is not a good approach for small log files. (It’s also important to note that S3 standard infrequent access and S3 one zone infrequent access storage both have a minimum billable object size of 128KB.)

Enabling Lifecycle Rules in CFML

If you decide that a lifecycle rule is the best way to automate changing storage classes for your files in S3, you can manage those rules (and change or delete them) from within your CFML applications. As always, the full code for this and all of my demos can be found in my AWSPlaybox GitHub repo.

There are five steps to enabling lifecycle rules via the AWS Java SDK:

Create a BucketLifecycleConfiguration object to apply to the whole bucket
Create a BucketLifecycleConfiguration$Rule object that defines a transition rule
Create a BucketLifecycleConfiguration$Transition object that specifies the storage class to change to after a specified period
Enable the rule
Apply the BucketLifecycleConfiguration to the bucket

Let’s look at this in code:

bucketLifecycleConfig = CreateObject('java', 'com.amazonaws.services.s3.model.BucketLifecycleConfiguration').init();
rule = CreateObject('java', 'com.amazonaws.services.s3.model.BucketLifecycleConfiguration$Rule').init();
// The rule Id just helps you to identify the purpose of the rule in plain text
rule.setId("Move to One Zone Infrequent Access after 30 days");
storageClassObj = CreateObject('java', 'com.amazonaws.services.s3.model.StorageClass');
transition = CreateObject('java', 'com.amazonaws.services.s3.model.BucketLifecycleConfiguration$Transition')
    .withDays(30)
    .withStorageClass(storageClassObj.valueOf('OneZoneInfrequentAccess'));
// setTransitions expects a Java List, which is just a CFML array
rule.setTransitions([transition]);
// Rules must be set to ENABLED to actually function on the bucket
rule.setStatus(bucketLifecycleConfig.ENABLED);
bucketLifecycleConfig.setRules([rule]);
s3.setBucketLifecycleConfiguration(s3BucketName, bucketLifecycleConfig);

The code follows the list of steps above, but there are a few points that bear explanation:

If you’re wondering what’s up with the $ in some of the Java class names, that’s how you instantiate nested Java classes in CFML. The Rule and Transition objects are nested classes of the BucketLifecycleConfiguration object in the AWS Java SDK, so you reference them with the $ in the class name. A big thanks to Ben Nadel for documeting that tip!
Rules are not automatically enabled when created or applied to a Transition object. They must be explicity ENABLEd to take affect. (This means you can also disable them as needed without deleting them off a bucket lifecycle configuration.)

Rules and Filters

While it’s easy to create a transition action for a whole bucket, you can only have one “all files” transition action per bucket. That is: you can only have one rule that applies to every file in the bucket. You can have many transition actions per bucket, but they must be defined with individual rules that apply filters to the files affected by the transition actions. You can filter by an object key prefix (the “path” to the object in the bucket), known as a Prefix Predicate, or a tag that’s applied to a file, known as a Tag Predicate. I’ll cover applying tags to files when you upload them to S3 in a future post.

Using both prefix predicates and tag predicates, you can have several rules for any given bucket and be specific about when different kinds of files in different parts of a bucket are automatically transitioned from one storage class to another. This is useful when you have long-established buckets with a lot of data and don’t want to incur the cost (and overhead) of moving the files into separate buckets to enforce different transition actions.

In the next post in this series, we’ll look at expiration actions, which are perfect for handling files with short lifespans, like log files. As always, if you have any questions about this, feel free to message me on the Twitter.