ChariotSolutions
ChariotSolutions
  • 671
  • 1 574 364
Optimizing Data Lakes: A Pipeline to Aggregate Small Files by Keith Gregory
Delivered at the Greater Philadelphia AWS User Group on February 8th, 2024. Presented by Keith Gregory, Chariot Solutions' Data Engineering & AWS Practice Lead. Hosted by Chariot Solutions.
Greater Philadelphia AWS User Group: www.meetup.com/gpawsug/
Chariot Solutions: chariotsolutions.com/
Overview:
Small files are the bane of a data lake, increasing your query times and processing costs. However, often you don’t get to control the data that you receive. For example, CloudTrail writes one file for each account and region, approximately every 15 minutes; dozens or even hundreds a day, some of which only have a few events.
An Athena query against the raw CloudTrail data might take minutes to execute, most of that time is due to the overhead of reading each file. By comparison, after aggregating the CloudTrail logs into one file per day, the same query takes only a few seconds.
In this talk, Keith Gregory walks through a data pipeline that uses Lambda to aggregate these files into a form that can be queried efficiently. He looks at the general design of such a pipeline, how to trigger it, how to monitor it, and how to be resilient to processing errors.
Переглядів: 69

Відео

Practical Insights for Navigating iOS Upgrades
Переглядів 525 місяців тому
Each year, Apple releases a litany of new features and APIs for developers to move into the future... usually by leaving some of the past behind. The transition from iOS 15 to iOS 16 was a particularly large jump: with major innovations like Swift Concurrency and the deprecation of the ubiquitous NavigationView, there were plenty of hurdles to clear in this update, but also many benefits to gai...
An Executive's Guide to Cloud Billing (and Strategies for Cost-Efficiency in 2024)
Переглядів 758 місяців тому
Gain a clearer understanding of your cloud bill, like never before. In this exclusive webinar presented by Chariot Solutions and Aimably, we'll delve into the nuances of cloud billing, empowering technology, finance, and operational leaders to make informed decisions in the cloud era. From understanding the unique aspects of cloud billing to interpreting your bill, pinpointing rationales for co...
There’s an Action for That! Exploring the Possibilities of GitHub Actions - Kedasha Kerr
Переглядів 2649 місяців тому
Did you know that you could order a pizza and send a text message with GitHub Actions? While Github Actions is the perfect tool for automating all aspects of your software workflows and deployment processes, you can also use actions to automate and streamline a variety of tasks. Whether you’re looking to compress images for the web, generate semantic release notes or check if your team member i...
TechChat Tuesdays #66: The DevOps of Sports Betting with Drew Rogers
Переглядів 6510 місяців тому
Today we talk to our own Drew Rogers about some of the nuanced aspects of Site Reliability Engineering (SRE) and Development Operations (DevOps) within the rapidly evolving domain of sports betting.
Let's Get Physical: Optimizing Big Data
Переглядів 12910 місяців тому
Fifty years of relational databases and online transaction processing have ingrained in us patterns of thought that do not translate neatly to the world of big data. Rather than focus on the traversal of individual entities, we must instead think in terms of large-scale set operations that exploit the parallelism of the underlying service. In this talk, Keith Gregory will look at performance op...
Leading With The Platform: A Platform Engineering Approach To Tech Debt - Jess Mink
Переглядів 26010 місяців тому
Platform teams, SREs, tool teams, DevOps…we are the stage crew of a software company: we’re only noticed when we mess up. This talk is about how platform teams have the potential to be proactive and lead projects across engineering so that things can be addressed they break, thereby reducing tech debt before it’s accrued, and they can be noticed for something positive for a change. We’ll be tal...
How To Steal From Maintainers - Richard Schneeman
Переглядів 68710 місяців тому
Maintainers all over the world are loaded. They’re loaded with tech debt, support requests, and a million things to do each and every day. In this talk, we’ll learn about actionable strategies for lightening their loads. And the beauty of the situation: they’ll never see it coming. We do this ‘one last job’ before I retire, and we’ll be rolling in the green stuff. That’s right. We’re talking op...
Demystifying Digital Twins: The Digital Twin Consortium’s Capabilities Periodic Table - Dan Isaacs
Переглядів 8510 місяців тому
A digital twin is a virtual representation of real-world entities and processes. Digital twin systems have the potential to transform businesses by accelerating holistic understanding, optimal decision-making, and effective action. Adopting digital twin systems can even help organizations achieve their sustainability goals. A digital twin of a healthcare facility rendered in 3D and using XR can...
Hacking The Pachyderm: Scaling Servers And People - Hazel Weakly
Переглядів 14010 місяців тому
Let’s set the stage, shall we? First, we have the starting cast: Kris Nova, Quintessence, and the plucky but nascent beginnings of the Hachyderm community. Then we have the stage: The Watertower, home of the beloved motly crew of Nova’s animaniac named servers, nestled lovingly in the basement. Next, we have the Hachyderm infrastructure team. Myself, Dominic, Malte, and Tani. And then, well, we...
Keynote - Choreorobotics: Teaching Robots How To Dance - Catie Cuan
Переглядів 49210 місяців тому
Choreographers and roboticists both organize moving bodies in space, yet the platforms and practices they employ are seemingly dissimilar. This talk describes the emerging field of choreorobotics, from dancing robots to novel programming interfaces, and why the intersection of these artistic and engineering fields is rapidly expanding. About Catie Cuan Catie Cuan is a choreographer, dancer, and...
Whither Programming Languages: Where Did They Go And Where Are They Going? - Jamie Allen
Переглядів 38610 місяців тому
Ten years ago, languages were a big part of the computing landscape, influencing a great deal of the way startups and enterprises tackled big challenges. The culture of an organization could almost be defined by the choice of language that they made to develop key solutions. But today, programming languages have become less prevalent, and new languages are no longer appearing. In this talk, I w...
A Survey Of JavaScript Meta-Frameworks - Ken Rimple
Переглядів 10610 місяців тому
Why you should consider a meta-framework for your React, Svelte or Vue project? In the dawn of the JavaScript front-end application development days, there were heavyweight development frameworks like Angular.js, Ember, and before that, jQuery. In those days, the framework had to do most of the heavy lifting, and developers used to back-end development looked for familiar patterns and features ...
WebXR For Cooperative Work, Thought, And Play - John Shaughnessy
Переглядів 5310 місяців тому
Engelbart had it right: The ways we work, play, learn, create, and think coevolve with our tools. The internet weaves itself into the fabric of our lives. It has changed how we gather and share information with each other. Personal computers, mobile devices, and now immersive devices shrink the distance between us and our digital identities. And yet, many of us find ourselves feeling overwhelme...
REPLs All The Way Up: A Rubric For Virtuous Feedback Loops - Avdi Grimm & Jessica Kerr
Переглядів 10610 місяців тому
We all know that the key to successful software development is iteration and feedback loops, whether that takes the form of the red-green-refactor cycle, or agile project planning. But not all feedback is equally valuable. How do you know if you’re poised to spin gold or just spin in circles? We propose a novel archetype for virtuous feedback cycles: the humble Read-Eval-Print-Loop (REPL). We’l...
dbt: A Software Engineering Approach To The Analytics Workflow - Kshitij Aranke & Brandon Segal
Переглядів 17610 місяців тому
dbt: A Software Engineering Approach To The Analytics Workflow - Kshitij Aranke & Brandon Segal
Reducing The Carbon Footprint Of Your Software Systems - Tammy McClellan
Переглядів 15510 місяців тому
Reducing The Carbon Footprint Of Your Software Systems - Tammy McClellan
Making Hoagies With Node, MongoDB, And Golang - Pi Mohan
Переглядів 8110 місяців тому
Making Hoagies With Node, MongoDB, And Golang - Pi Mohan
Your Browser is Pooped: Using Web Workers To Create A Browser-Based Barcode Scanner - Drew DeCarme
Переглядів 10510 місяців тому
Your Browser is Pooped: Using Web Workers To Create A Browser-Based Barcode Scanner - Drew DeCarme
An Open Book Test: Securing Open Source Software With OpenSSF’s Scorecard - Stephen Augustus
Переглядів 7210 місяців тому
An Open Book Test: Securing Open Source Software With OpenSSF’s Scorecard - Stephen Augustus
Building And Deploying Omni-Channel Digital Experiences At Scale - Russ Danner
Переглядів 7510 місяців тому
Building And Deploying Omni-Channel Digital Experiences At Scale - Russ Danner
Navigating A World Where AI-Generated Misinformation Can Be Created At Scale - Alex Cui
Переглядів 9210 місяців тому
Navigating A World Where AI-Generated Misinformation Can Be Created At Scale - Alex Cui
Tips For Detecting The Use Of AI-Generated Text - Dr. Jake Ryland Williams
Переглядів 13610 місяців тому
Tips For Detecting The Use Of AI-Generated Text - Dr. Jake Ryland Williams
Shaping Up Your Development Process With Shape Up - Martin Snyder
Переглядів 19710 місяців тому
Shaping Up Your Development Process With Shape Up - Martin Snyder
Achieving Resiliency With A Multi-Cloud Or Multi-Region Approach - DaShaun Carter
Переглядів 11710 місяців тому
Achieving Resiliency With A Multi-Cloud Or Multi-Region Approach - DaShaun Carter
Calling Functions Across Languages - Richard Feldman
Переглядів 1,4 тис.10 місяців тому
Calling Functions Across Languages - Richard Feldman
Starbeam: Universal Reactivity And What It Means For You - Yehuda Katz
Переглядів 10710 місяців тому
Starbeam: Universal Reactivity And What It Means For You - Yehuda Katz
Lightning Talk: A Simple Talent Development Program for Software Departments - Sue Gerace
Переглядів 3410 місяців тому
Lightning Talk: A Simple Talent Development Program for Software Departments - Sue Gerace
Lightning Talk: Linux is Getting Rusty (And So Is Android) - Matt Gilbride
Переглядів 4110 місяців тому
Lightning Talk: Linux is Getting Rusty (And So Is Android) - Matt Gilbride
Lightning Talk: The Art of Problem-Solving: Strategies for Finding Your Way in the Dark-Gary Sieling
Переглядів 14210 місяців тому
Lightning Talk: The Art of Problem-Solving: Strategies for Finding Your Way in the Dark-Gary Sieling

КОМЕНТАРІ

  • @gofudgeyourselves9024
    @gofudgeyourselves9024 Місяць тому

    35 years remaining

  • @conundrum2u
    @conundrum2u Місяць тому

    "why can't it *just* be this way" well you see.... there's over 40 years of history behind that reason and many architectures in-between. I look forward to new programming languages in 20 years where a young language designer goes "why didn't they *just* do it this way in the 2020's??!?!?!?" Zig is interesting. We'll see how it goes.

  • @hyperbolee1060
    @hyperbolee1060 2 місяці тому

    So little information in this drag of a talk

  • @joshuaclayton6940
    @joshuaclayton6940 2 місяці тому

    I love the involuntary case study of the slideshow breaking just as Andrew was making a joke about software being broken.

  • @roozbehvalavi8700
    @roozbehvalavi8700 2 місяці тому

    The worst start of a presentation ever!! and too biased! although I really enjoyed the Rust part!

  • @youganon
    @youganon 2 місяці тому

    But how does FP fulfil that game programmer's idea of "Design software based on hardware, so that it runs fast"?

  • @kyleschlicht4800
    @kyleschlicht4800 3 місяці тому

    Disruption to the lMAX!!!

  • @ciCCapROSTi
    @ciCCapROSTi 3 місяці тому

    What a long and unnecessary intro. This woman is annoying as hell.

  • @ciCCapROSTi
    @ciCCapROSTi 3 місяці тому

    I really want to know what Zig brings to the table over a feature-restricted C++. You can do most, if not all of these things in C++, and you don't HAVE TO use the nasty stuff.

  • @capability-snob
    @capability-snob 3 місяці тому

    David built a capability system? I mean, I think that Digital Key app would make Marc Steigler proud.

  • @BloodnutXcom
    @BloodnutXcom 3 місяці тому

    I'm not sure, aside from compiling to wasm, what this language innovates on? All these features existed in Scala for example for absolute ages, especially the "I haven't seen it anywhere" Backpassing, which existed in Scala for years and is called for-comprehension. The syntax is just identical.

  • @marcosissler
    @marcosissler 4 місяці тому

    Oh my God! What a great talk and presentation. My knowledge of Go has now expanded with two more releases! I am surprised at how powerful the interface is. Thank you for sharing this. I will spread this at the Brazilian community here. Hope see more soon and for sure I will take a look at your book. Bests, Marcos. Ow! How do you do your presentation with the run button? I will see if I can figure out at your presentation link, if there is one. :)

  • @suma_riya
    @suma_riya 4 місяці тому

    Hope you are well. I am sincerely sorry to disturb you. Your videos are fine but your videos are not getting views. Moreover, the number of subscribers to your UA-cam channel is also very low. I checked out your UA-cam channel. I found some problems. 1. SEO Score is very low 2. No Title - Description - Tag SEO friendly 3. No Share Social Media Platform and many problems on your UA-cam Channel. If you do proper SEO of your videos and SEO of your channel, your videos will go viral very quickly. You can generate a good view. Hope you understand my point. I am always ready to solve your problem.

  • @josephs1732
    @josephs1732 4 місяці тому

    the laughing lady definitely likes Stuart lol

  • @adanjsuarez
    @adanjsuarez 4 місяці тому

    Zig is a monster!... in a very good way.

  • @edgeeffect
    @edgeeffect 5 місяців тому

    I'm SO pleased that there's finally someone out there among the "movers and shakers" throwing a little shade at the C preprocessor... I've been quietly hating it for about 25 years now.

  • @anushaakkerapu848
    @anushaakkerapu848 5 місяців тому

    🎉🎉 thank you this tutorial

  • @saschadibbern339
    @saschadibbern339 5 місяців тому

    Found out about htmx two months ago. Building a htmx app after not touching webdev for over 10 years. I am happy not to have wasted so much time on SPA frameworks. I feel so productive 😊

  •  5 місяців тому

    Can there be something like a Virtual Dom for htmx? So you do react-like Dom manipulation on the server and that automatically gets transmitted to the browser.

    • @Leeway4434
      @Leeway4434 3 місяці тому

      you dont need a virtual dom with htmx. you change the real dom using the hypermedia returned from the web server. dom manipulation on the server is templating/embedding data from your db into html

  • @lockbert99
    @lockbert99 5 місяців тому

    What are the 6 of 10 most popular programming languages (3 years ago) that are either C++ or a descendant of C++? Does being a descendant just mean that it has curly braces for block delimiters and variable type before variable name? That makes it a descendant of C. So it has that plus classes?

  • @SV-tc8cu
    @SV-tc8cu 5 місяців тому

    it is helpful but could be better. a lot of areas to improve and clarify for go beginners

  • @charlesuneze4920
    @charlesuneze4920 6 місяців тому

    Top-notch content

  • @JohnMatthew1
    @JohnMatthew1 6 місяців тому

    excellent!

  • @blacky7801
    @blacky7801 6 місяців тому

    so this language has tag unions, which are not to be confused with tagged unions. Except they also can be tagged unions, because the tags may have associated types. got it

  • @achrefab3142
    @achrefab3142 6 місяців тому

    Man, you saved my day! 🫡

  • @Estereos
    @Estereos 6 місяців тому

    eliminate also "error handling" and maybe then we will talk about how good your language is... only idiotic program "handles" errors, uses asserts, prints out the stack trace. Good program deal with errors, there should NOT be a condition when program doesn't know what to do. even f*cking "division by zero"

  • @deadmarshal
    @deadmarshal 6 місяців тому

    Add more buzzwords lol

  • @0LoneTech
    @0LoneTech 7 місяців тому

    Why are arguments comma separated in function definition? It seems painfully different from function application.

  • @fifty6737
    @fifty6737 7 місяців тому

    many people in the comment doesn't really get how awesome a language like this means i remember React.js introduction, nobody wanted it, now react & js framework that came to improve upon it run the web C is so horribly bad and weird that it's impossible to maintain, easier to make buggy and unsafe code with it Zig is a language with big balls, it's doing the obvious things that nobody did for decades - Why have build systems that are in another obscure language when you can use the language itself to handle the building - Macros are weird, why not program the compile time stuff with the same language - Generics are compile time stuff, again why create weird syntax for generic typing when you can just program that type in compile time with the same language - allocation and cleanup should always be next to each other - why have some elaborate typing system for exceptions, when an enum for the different kind of errors is much easier to deal with - unit tests are boring and complicated because they are always an afterthought and not made from the get go to be with language, zig does unit testing better than 100% of the languages - std = @import("std") makes so much fucking sense - and zig acknowledge that C have huge codebase, so they built a better tooling for c than c itself has, and you can import header files natively i like Rust, but now i love Zig more, zig have the balls to create a proper language that is not complex like Rust, and doesn't hold your hand and force some set of rules upon you like Rust Borrow Checker But zig has a long way to be mature enough and feature rich

  • @belsheikh
    @belsheikh 7 місяців тому

    @16:25 I think what Dan meant when he said: "we had the OPPORTUNITY.." is that rebuilding the application again was something easily done by oop not easily done in other applications written in previous paradigms making it experimental a good thing not a bad thing.

  • @terragame5836
    @terragame5836 7 місяців тому

    My problem with non-native UIs is that they often lack some behaviours and capabilities of native ones. For example, the Java UI file selection dialog doesn't let me perform a right-click action on a folder before selecting it, as well as a lot of other things I would've been able to do with a native equivalent. Some cross-platform multi-windows come with a horrible drag lag due to not issuing a redraw update every tick. They also often don't support half-screen docking by dragging the window to the side. Qt scrollbar, if I recall correctly, misses the feature of the native one that if you drag too far in the perpendicular direction, it wouldn't move at all, preventing an accidental mistake in scroll direction. There's a billion little things like this that just cannot be accounted for by a single ultimate cross-platform UI framework. BUT webpage-based UI seem to have become a nice compromise. A browser provides a sort of 'frontend' (as in 'compiler fronted', i.e. a 'common representation -> native result' transformation), which more often than not relies on native UI components, or at least mimics their behaviour well enough for every platform individually. And the developers get a relatively uniform target representation. So, as much as I dislike javascript and the tendencies it has caused in webapps, I think webpage-based cross-platform UI is a step forward from whatever Java, Qt & etc. tried to do

  • @ms77grz
    @ms77grz 7 місяців тому

    👍👍

  • @theodorosgeorgitsis5600
    @theodorosgeorgitsis5600 7 місяців тому

    This was one chochsky too much fr me. Did not get shit :D

  • @arjunkandaswamy1532
    @arjunkandaswamy1532 7 місяців тому

    how RUST is better than ZIG in language design

  • @kahnfatman
    @kahnfatman 7 місяців тому

    The dialectics goes on: Thesis -> Antithesis -> Synthesis (which itself becomes a new Thesis)

  • @klirmio21
    @klirmio21 7 місяців тому

    No offense but. I like Richard Feldman, he is very good presenter. He had been working for a long time with Elm, now he decided to create his own functional programming language. How is Roc different than Elm?

  • @jpenneymrcoin6851
    @jpenneymrcoin6851 7 місяців тому

    "the software must not harm or fail" HAHAHAHAHAHAHAHA I so wish the industry operated with this viewpoint. Whoo. How about : "the software must not cost anything and all errors must be hidden from upper management"?

  • @lockbert99
    @lockbert99 8 місяців тому

    11:27 Looks like a bare "greeting" should be info.greeting

  • @aftalavera
    @aftalavera 8 місяців тому

    Keep repeating it! Someone may believe it!

  • @davidlewis4683
    @davidlewis4683 8 місяців тому

    Literally just dropped in to add a like.

  • @BStack
    @BStack 8 місяців тому

    Great talk it definitely leveled my understanding of interfaces. And +1 for any talk that shows how the subject can be used when testing.

  • @BryanChance
    @BryanChance 8 місяців тому

    Awesome talk..i'm in!!!

  • @julianavar3836
    @julianavar3836 8 місяців тому

    Also, remember that in Unity (11:00) you, the developer, must pay per user install :D So convenient

  • @elenagavrilova3109
    @elenagavrilova3109 8 місяців тому

    70% of speech could be left for your psy. therapist, this reflection has to do a lot w poor communication skills and toxic env at your office and does not answer the question how would you test it.

  • @gJonii
    @gJonii 9 місяців тому

    I got unnecessarily distracted wondering why all these people refer to themselves with third person pronouns.

  • @Karakatiza666
    @Karakatiza666 9 місяців тому

    Great talk. LiveScript (untyped, transpiled into JS) has backpassing (called backcall)

  • @drew7537
    @drew7537 9 місяців тому

    Great talk. I'm completely htmx-pilled, and excited to see where things go for the framework.

  • @trustingod0
    @trustingod0 9 місяців тому

    Great Job. I learned so much from your presentation.

  • @KManAbout
    @KManAbout 9 місяців тому

    okay i am a little confused why you would use react native with clojurescript rather than just clojure. since that can run natively on andriod is it because this is a crossplatform app ?

    • @David-iq1kd
      @David-iq1kd 7 місяців тому

      I don't believe Clojure runs natively on Android - if it does how do you go about it?

    • @KManAbout
      @KManAbout 7 місяців тому

      @@David-iq1kd Clojure can run natively since it uses the jvm

  • @nicolemcdonald2298
    @nicolemcdonald2298 9 місяців тому

    'promo sm'