Building a near real-time data warehouse using AWS technologies & Apache Velocity

This video explains how to directly integrate AWS API Gateway with Kinesis Firehose & build a near real-time Cloud Data Warehouse using Apache Velocity.


PutRecordBatch Documentation Link:
------------------------------------------------------------------------
https://docs.aws.amazon.com/firehose/...

API Gateway mapping template :
--------------------------------------------------------------------
https://docs.aws.amazon.com/apigatewa...

Reference About Apache Velocity:
-------------------------------------------------------------
https://velocity.apache.org/engine/de...


Code used in this video are available in the pinned comment.

API Gateway - Kinesis Firehose Mapping Template:
---------------------------------------------------------------------------------------
#set($context.requestOverride.header.X-Amz-Target = "Firehose_20150804.PutRecordBatch")
#set($context.requestOverride.header.Content-Type = "application/x-amz-json-1.1")                
#set($inputRoot = $input.path('$.Events'))
{ "DeliveryStreamName":$input.json('$.Firehose_name'),
 "Records": [
   #foreach($elem in $inputRoot)
   {
     "Data": "$util.base64Encode($input.json("$.Events[$foreach.index]"))"
   }#if($foreach.hasNext),#end
   #end
 ]
}

Check this playlist for more Data Engineering related videos:
Demystifying Data Engineering with Cl...

Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
https://doc.clickup.com/37466271/d/h/...

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL

• #science_technology
• #building_a_near_real_time_data_warehouse_using_aws_technologies_apache_velocity
• #api_gateway_mapping_template
• #api_gateway_mapping_template_and_access_logging_variable_reference
• #real_time_data_warehouse_using_aws_technologies
• #build_and_optimize_a_real_time_stream_processing_pipeline_with_amazon_kinesis
• #building_a_data_platform_with_aws_apache_software_foundation