Nebulex, a local and distributed caching toolkit for Elixir.

Like with many other caching toolkits, Nebulex covers all the basics.

This article is not a deep dive into Nebulex features but an example of how I used it in my own application.

Nebulex Basics

Inserting (with time to live):

Blog.Cache.put(key, data, ttl: :timer.hours(1))

Retrieving:

Blog.Cache.get(key)

Counters:

Blog.Cache.incr(:my_counter)

Blog.Cache.incr(:my_counter, 5)

Deleting:

Blog.Cache.delete(1)

Now, this is a partial list of Nebulex capabilities. There are several more functions to interact with the cache, covering many more use cases you might have.

But Why Choose Nebulex?

While looking at Nebulex, two things caught my eye that would make Nebulex easily fit into any Phoenix implication (in my opinion).

1. The easy setup using a mix task

2. Nebulex implementation of Cache as System of Record

Easy Setup

Starting with Nebulex is extremely easy because of their handy dandy mix task. Just run:

mix nbx.gen.cache -c Blog.Cache

This command will generate the cache configuration in your config.exs and create a cache module. The last step is to add the cache module to your supervision tree. This step is called out in the docs and task output.

Cache as System of Record (Cache-as-SoR)

This feature caught my eye and is perfect for how I want to use the cache.

If you need to learn what Cache as System of Record is, here is an excerpt from the Nebulex docs.

The cache-as-SoR pattern implies using the cache as though it were the primary system-of-record (SoR).

The pattern delegates SoR reading and writing activities to the cache, so that application code is (at least directly) absolved of this responsibility. To implement the cache-as-SoR pattern, use a combination of the following read and write patterns:

• Read-through

• Write-through

Advantages of using the cache-as-SoR pattern are:

• Less cluttered application code (improved maintainability through centralized SoR read/write operations)

• Choice of write-through or write-behind strategies on a per-cache basis

• Allows the cache to solve the thundering-herd problem

To better understand what this means, here is a code example of implementing a read-through pattern in my project:

  @decorate cacheable(
              cache: Cache,
              key: {StreamSettings, user_id}
            )
  def get_stream_settings_by_user_id(user_id) do
    Repo.get_by(StreamSettings, user_id: user_id)
  end

In my Settings module, the get_stream_settings_by_user_id is called every time a caption payload is broadcasted to viewers to check if the broadcaster wants to add additional profanity filters or other text modifiers. Nebulex offers the cacheable decorator function that will cache the result of my get_stream_settings_by_user_id function using a specified key. If data already exists in the cache and hasn't expired, it will return the cached result the next time this function is called.

In the above code, I am caching StreamSettings by the user_id and saving time by avoiding trips to the database till the cache entry expires. I love how easy it is to set up caching on any function I use.

But what happens if the user's StreamSettings are updated? That's where the write-through pattern's decorator comes in! Let's look at some code again:

@decorate cache_put(
            cache: Cache,
            key: {StreamSettings, stream_settings.user_id}
          )
def update_stream_settings(%StreamSettings{} = stream_settings, attrs) do
  stream_settings
  |> StreamSettings.update_changeset(attrs)
  |> Repo.update()
end

cache_put is the magic function here. It will update the cache with the result of update_stream_settings using the key specified, {StreamSettings, stream_settings.user_id}. I like how Nebulex makes it simple to wrap your existing functions with cache reads and writes.

But the value you are caching in your read-through is different from the value you are caching in your write-through function!?!?

Good Catch! Well, Nebulex also has a function for that. If the value you are caching using the cacheable function is a different shape from the value that the cache_put function, you can use the cache_evict function. Let's go through an example:

# Cache - key:{StreamSettings, 123}, value: nil
get_stream_settings_by_user_id(123) 
# Return: %StreamSettings{some_data}
# Cache - key:{StreamSettings, 123}, value: %StreamSettings{some_data}

Calling get_stream_settings_by_user_id places takes the returned value and places it in the cache, going from nil --> %StreamSettings{some_data}

update_stream_settings(stream_settings, %{update_data}) 
# Return: {:ok, %StreamSettings{some_updated_data}}
# Cache - key: {StreamSettings, 123}, value: {:ok, %StreamSettings{some_updated_data}}

But the problem arises with the return value of update_stream_settings. Instead of the result being a StreamSettings struct, it instead returns a tuple {:ok, %StreamSettings{}}. But cache_evict is here to save the day!

@decorate cache_evict( # <------- The change -------
            cache: Cache,
            key: {StreamSettings, stream_settings.user_id}
          )
def update_stream_settings(%StreamSettings{} = stream_settings, attrs) do
  stream_settings
  |> StreamSettings.update_changeset(attrs)
  |> Repo.update()
end
# Cache - key:{StreamSettings, 123}, value: nil

Instead of caching the new result from update_stream_settings this will remove the value from the cache, removing the worry or mismatched cached data and instead letting the read-through function, aka cacheable write to the action on the following invocation.

Conclusion

Nebulex's Cache as System of Record is a feature that fits right into my mental model of how I would like to work with a cache. Just decorate the functions I use throughout my applications and BAM caching with minimal configuration. Nebulex has more features; look at Nebulex if you're in the market for a caching toolkit.

Happy Coding!

But over time, I discovered Phoenix Channels has a downside. If you use long-lived connections, memory usage on your server will remain unusually high.

How I Use Phoenix Channels

Now for a bit of background on how I use Phoenix Channels. My Stream Closed Captioner application is a free Closed Caption service I offer for Twitch streamers and Zoom meetings. Users log into a dashboard on the website and turn on Speech-to-text. Then a Phoenix Channel sends Text from the front end to the back end, where it goes through a couple of filters like profanity filtering, among other things.

If a Twitch streamer is using the service, it will broadcast the resulting text over another Phoenix Channel for that user's viewers to receive. If they use Zoom, it will send an HTTP request to the Zoom meeting's Closed Caption endpoint.

On the Twitch viewer side, the app communicates to the server via GraphQL. When a Twitch streamer goes live, the viewer client will initialize a GraphQL Connection (aka WebSocket/Phoenix Channel) to receive broadcaster captions from the server.

In other words, each streamer will have a connection to send captions to the backend. Each viewer is another connection that lasts for the live broadcast. For example, if a live stream has 200 viewers, that makes for 200 + 1 connections.

What Does That Mean For Memory Consumption?

Typically, Twitch live streams last at least 3 hours, and I quickly discovered these connections can take up a lot of memory. 6-7 gigabytes of memory to support the number of connections necessary for all streams... YIKES

But why? I spent a little time with my nose in some books and found this passage in
Real-Time Phoenix:

Short-Life Versus Long-Life Processes

Real-time applications rely on long-lived processes in order to have a direct connection to clients—this allows them to send data immediately when the data is available. There is a downside to this, though. It’s possible that processes get stuck in a state where garbage collection doesn’t occur, but memory isn’t being used. This can cause large memory bloat when amplified across thousands of processes.

If a piece of memory makes it past generational garbage collection, it lives until a full-sweep garbage collection occurs. This happens, by default, every 65535 generational passes or when the process is close to using its available memory. It’s possible for a Channel, Socket, or other long-lived process to get stuck in a state where there is plenty of free memory, but not enough work to trigger a generational pass. A process in this state will live without memory being collected, and it will potentially take up more memory than necessary.

Processes that are created and terminated quickly (short-lived processes) do not get into this state because their memory is reclaimed when the process is terminated. While all applications use long-life processes to some degree, real-time apps use them in the thousands. This amplifies the impact of memory bloat.

You can fix this problem by forcing a full-sweep garbage collection to occur...

Why memory consumption is so high now makes a lot more sense. Every connection that is being initialized is long-lived, and the garage collector won't clean up the memory of these processes until 65k garbage collection sweeps have happened. Now what?

But There Is A Solution!

Luckily, Real-Time Phoenix also recommends a few things you can do to force garbage collection sooner.

One such setting is ERL_FULLSWEEP_AFTER.

Understanding ERL_FULLSWEEP_AFTER

ERL_FULLSWEEP_AFTER is a setting related to the Erlang Virtual Machine’s (VM) garbage collection mechanism. It allows you to specify the maximum number of generational collections before forcing a full-sweep, a type of garbage collection that cleans up memory that is no longer in use. By default, this is set to 65535.

ERL_FULLSWEEP_AFTER 20

Real-Time Phoenix author mentioned that they had success setting this value to 20 for their application, so this is what I set my environment variable, but it can be any value that works for you.

The Result!

Within minutes of setting the ERL_FULLSWEEP_AFTER environment variable to 20, I saw results in my Gigalixir memory consumption dashboard. I went from using 6 gigabytes of memory down to around 3.5 gigabytes in a dramatic fashion. Unfortunately, I forgot to snap a screenshot of the dashboard, but the screenshots below still show the updated setting at work.

The above image shows memory consumption after two deployments. You can see the memory consumption jump up to my historical average of around 6 gigabytes after all existing users on the application reconnect to the new instance. After the 20 minor garbage collections kick in, the major garbage collector is triggered, and you see a significant drop.

In this image, those spikes are Twitch streamers going live and connections for all their viewers being made. You can quickly see how much memory connections can take up. But after 20 minor garbage collections, the major garbage collector starts cleaning things up, and you see a significant drop. Amazing! With one simple setting, I solved my memory consumption woes without issues.

Conclusion

If you're building a real-time application using Phoenix Channels, it's essential to be mindful of memory usage, especially if you're dealing with long-lived connections. While Phoenix Channels are fantastic for using WebSocket connections, they can cause memory bloat when used in thousands of processes. However, with a bit of knowledge and tweaking of the ERL_FULLSWEEP_AFTER setting, you can force garbage collection sooner and dramatically reduce memory consumption. So, if you're experiencing a similar issue with your Phoenix Channels application, give this solution a try and see the results for yourself.

I would also like to make a plug for Real-Time Phoenix by Stephen Bussey. It's an awesome book, and you can find the passage referenced in the chapter Manage Your Application’s Memory Effectively, along with more goodies.

Happy coding!

Supporting resource for my talk at GraphQL Summit, "Understanding GraphQL Security: Protecting Your Server from Abuse" Resources

GraphQL server features to limit or disable in production

• Disable Introspection - Not only disable GraphiQL but also prevent introspection.

• Block field suggestions - Prevent returning field suggestions and leaking your schema to unauthorized actors.

• Character Limit - Limit the number of characters in a GraphQL query document.

• Cost limit - Limit the complexity of a GraphQL document.

• Max Aliases - Limit the number of aliases in a GraphQL document.

• Max Depth - Limit the depth of a GraphQL document.

• Max Directives - Limit the number of directives in a GraphQL document.

Learning Resource

• GraphQL Cheat Sheet - Learning resource for possible vulnerabilities attack vectors

• Black Hat GraphQL

  

Language Specific Solutions

NodeJS

• GraphQL Armor - Add a security layer to Apollo Server, GraphQL Yoga, Envelop

Elixir

• Safety Limits - Absinthe configurations to help enforce cost and token limits

• Vigil - Disallows introspection of a GraphQL schema and sanitizes error messages to be completely generic and not reveal information about your schema.

• AbsintheSecurity - provides utilities to improve the security posture of APIs built with Absinthe GraphQL.

Ruby GraphQL

• How to set max_depth and max_complexity to apply limits to incoming queries.

• Disable introspection

How to activate the GitHub Command Palette

The GitHub Command Palette is available as a feature preview, which means you can opt-in to try it out before it becomes generally available. To activate the GitHub Command Palette, follow these steps:

1. Go to https://github.com/

2. Click on your top-right profile icon

3. Select "Feature preview."

   

4. Under "GitHub Command Palette," click "Enable."

   

5. You'll see a confirmation message that says, "GitHub Command Palette enabled."

6. You can now use the GitHub Command Palette on any page on GitHub

How to open the GitHub Command Palette

To open the GitHub Command Palette, use one of the following default keyboard shortcuts:

• Windows and Linux: Ctrl + K or Ctrl + Alt + K

• Mac: Command + K or Command + Option + K

You can also customize the keyboard shortcuts in the Accessibility section of your user settings. For more information, see "Customizing your GitHub Command Palette keyboard shortcuts."

When you open the GitHub Command Palette, it shows your location at the top left and uses it as the scope for suggestions. For example, if you're on a repository page, it will show suggestions related to that repository.

How to navigate with the GitHub Command Palette

You can use the GitHub Command Palette to navigate to any page you can access on GitHub. For example, you can quickly jump to your organizations, repositories, issues, pull requests, projects, files, and more.

When you open the GitHub Command Palette, the suggestions are optimized to give you easy access to top-level pages like the Issues page or Pull requests page. If the location you want isn't listed, start entering the name or number for the location to refine the suggestions.

You can also use modes to find specific types of resources and execute commands. To enter a mode, type one of the following symbols:

- > Enter command mode
- # Search for issues, pull requests, discussions, and projects
- ! Search for projects
- @ Search for users, organizations, and repositories
- / Search for files within a repository scope

For example, if you want to find an issue by its number, type # followed by the issue number and press Enter.

Example of Supercharging Your Development Workflow

If you want to check the status of a pull request you reviewed earlier but don't remember the number or the title. You can use the GitHub Command Palette to find it quickly.

Press Ctrl + K (Windows/Linux) or Command + K (Mac) to open the GitHub Command Palette. Type # and see a list of issues, pull requests, discussions, and projects you have recently interacted with. You can type a few keywords that you remember from the pull request and see the matching results. Select the one that you are looking for and press Enter. You will be taken to the pull request page to see the latest comments and changes.

Boom, navigating GitHub efficiently just happened.

Conclusion

Give GitHub Command Palette a try and see if it helps improve your developer workflow if you find yourself navigating GitHub repos on a regular basis.

Animated Drawings is a web-based application that lets you upload your own drawings of human-like figures or choose from some demo figures and then select from various preset animations such as dance, funny, jumping, and walking. The tool uses advanced AI techniques to capture, rig, deform, and animate your drawings in seconds.

You can try it out for yourself here: https://sketch.metademolab.com/

Here are some tips to get the best results from Animated Drawings:

• Make sure your drawing is clear and simple. Avoid using too many colors, details, or background elements that might confuse the tool.

• Draw your figure with two arms, two legs, a head, and a torso. The tool works best with human-like characters that have these body parts.

• Adjust the capture box to fit snugly around your drawing. Make sure to use the pen and eraser tools to select regions that were missed during auto masking or remove unwanted areas before animating. This will make your animation look even better.

• Experiment with different animations.

• Remember to have fun. It’s free, after all.

Since I am horrible at drawing, I thought, "Why not get DALL-E to draw me something and use AI to animate my AI-created drawing." Here is the photo I decided to use:

I used the prompt: "Children style drawing, white background, goose spreading wings."

And here is the end result using the "waving hello" animation.

Meta AI has open-sourced the project and encourages developers to create new, immersive experiences with drawing-to-animation. You can learn more about the project and access the code and data here: https://github.com/facebookresearch/AnimatedDrawings

I think Animated Drawings is an excellent example of how AI can inspire creativity and joy. What do you think? Have you tried it out? Feel free to share your creations in the comment below.

Bing Image Creator is powered by Dall-E 2, an AI tool developed by OpenAI that can create images based on text prompts. You can use Bing Image Creator to explore your creativity, illustrate your ideas, or just have fun with AI.

The weird part about this new feature is that when visiting Bing.com you won't see any indications that this feature exists but momentarily I will show you exactly how to use it.

How to use Bing Image Creator

There are two ways to use Bing Image Creator.

1. Through Bing Chat

2. Dedicated Image Creator website

Using It In The Bing Chat

To use Bing Image Creator through the Bing chat window, you need to have access to Bing Preview, a new version of Bing that features an AI chat functionality. You can sign up for the Bing Preview by going to bing.com/chat in Microsoft Edge and signing in with your Microsoft account.

If you do already have access and are on the “Chat” page. Then, you need to change the conversation style to Creative using the selection box.

Next, you can enter a prompt into the chat window for what you'd like your image to be. Make sure you specify somehow that you'd like an image created. For example, you can type "draw me a picture of a dragon" or "create an image of a cat wearing sunglasses". Hit enter and see what Bing Image Creator comes up with. It will make 4 different interpretations of your image and use up one of your "boosts" (Find out about what boosts are in the Image Creator section).

You can make further adjustments from there using additional prompts which will use up your "boosts".

The annoying aspect about using the Image Creator through Bing Chat is that it hides the concept of "boosts" and you can easily run out of them as you ask it to generate more images or make edits.

Here is what it created for me using the prompt "draw me a goose wearing aviators, cartoon"

Using The Dedicated Image Creator

To use Bing Image Creator through the dedicated website, you don't need to have access to Bing Preview or use Microsoft Edge. You can simply go to bing.com/create and sign in with your Microsoft account.

Starting you will have a set amount of “boosts” to have “priority” image generation. But fear not, you can still generate more images for free after you use up all your "boosts" but at a slower rate.

Once you have Image Creator open, you can enter a prompt into the text box for what you'd like your image to be. You can also provide additional contexts like location or activity, and choose an art style like you would in the Bing Chat. Click Create and see what Bing Image Creator generates.

Here is what was generated for me using the following detailed prompt "Looking down a downtown street with sky scraper on each side of the street and the sunset in the distance over the water, pixel art"

Bing Chat vs Image Creator

Although you are essentially using the same product, the "Image Creator". One observation from using the Image Creator directly versus the Bing Chat is that adjusting the image generation using the Image Creator isn't as easy as the Bing Chat follow-up prompts. You will need to completely adjust your prompt with trial and error. Also, Bing Chat gives you some great follow-up prompts to give you ideas on what else you can do.

Here is how easy it was to improve the image using the Bing Chat with 2 follow-up prompts.

"Make it more details with more of the water visible and the sun half way sunset.

Then, "Add cars on the street and seagulls flying in the distance silouette by the sun."

But one nice thing Image Creator is that you are not forced to use the Edge browser if you find that to be an annoyance.

Tips and Tricks

Here are some tips and tricks for using Bing Image Creator effectively:

• Be specific and descriptive with your prompts. The more details you provide, the more likely you are to get an image that matches your expectations. For example, instead of typing "a dog", try typing "a brown Labrador retriever sitting on a couch".

• Experiment with different prompts and art styles. You can try different combinations of words and styles to see how they affect the output. For example, you can type "a dragon made of flowers" and choose "realistic" or "abstract" as the art style.

• Use quotation marks for exact phrases. If you want Bing Image Creator to use a specific phrase as part of your prompt, put it in quotation marks. For example, if you type "a poster for \"The Matrix\"", Bing Image Creator will try to create an image that resembles a poster for the movie The Matrix.

• Don't worry about saving your creations, you can see everything you created in both Bing Chat and Image Creator by visiting the Image Creator dedicated page and clicking on the "Creations" tab

Final Thoughts

Bing Image Creator is a great way to unleash your imagination and see what AI can do with little commitment. Try it out today and share your creations with us!

It's a tie for my favorite between these two, click them for direct links to them in Image Creator:

React architecture is flexible and does not enforce any particular pattern to write a complex application. Developers are free to choose the design pattern of their choice. React architecture is built on a solid foundation and is simple, flexible, and extensible.

In other words, React is a library that can be used to build complex applications with the design pattern of your choice. The primary piece to this change is React Server Components, which were teased a couple of years ago. React Server Components are components that can be rendered on both the server and the client. They allow developers to build applications that span both sides of the application, combining the rich interactivity of client-side apps with the improved performance of traditional server rendering. The new documentation now reflects React as a library to use on both the client and server sides. It pushes single-page apps as a footnote and now recommends using a framework like Next.js that incorporates React Server Components instead of create-react-app.

The React team has admitted that they need to do a better job of adequately communicating the initiatives they have going on. We were teased with React Server Components and other features like Suspense a couple of years ago, but there was mostly radio silence after that until the new docs were released, making it appear like React took a hard right turn off the SPAs highway. But React has been silently heading in this new "direction" for a while now; it just wasn't noticeable.

Is React no longer for SPAs? NO, it’s not only for SPAs but also for the server. The React team wants you to start considering React in that way so you can get the best performance possible for your users by first considering frameworks that use the newer features of React to their fullest extent. But in the end, the beauty of “React Architecture” is that you can choose how to use React.

Sorry about the mouse cursor not showing up in this video! If you're not already using the GitHub Copilot Labs extension for VSCode then you missing out on some awesome features. In this video I show you one feature that blew my mind.

You can install Github Copilot Labs here https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-labs

Find out all the details about Github Copilot Labs Here: https://githubnext.com/projects/copilot-labs/#how-to-use-copilot-labs

But first, if you're not sure what RedwoodJS is, it's an open-source full-stack web framework that makes building applications extremely easy. Think of Ruby on Rails, but all in JavaScript using libraries you might already be familiar with and extremely easy to set up. You can check it out here https://redwoodjs.com/docs/introduction.

RedwoodJS is easy to set up and start, but what would make it even better? Making it extremely easy to get your developer environment set up and running with a couple of clicks and compatible with Github Codespaces. This short guide will give you a template needed to do just that!

We will use the Visual Studio Code Dev Container extension/feature to accomplish this goal. It's a fantastic tool I use almost daily to run my projects in what I like to think of as a development sandbox. Since each project runs in its own sandbox, it doesn't have to worry about installing different Ruby, Rails, NodeJS, Elixir, or Erlang to get my project up and running on my computer. So let's get to it.

Required To Get Started

• Docker for Desktop
• Visual Studio Code

Set Up Dev Container!

Alright, to get started, open up your RedwoodJS project in VSCode. If you don't have one, it's as simple as.

yarn create redwood-app ./super-awesome-project-name

Now let's install the Remote Development extension. It should look like the following image.

The Remote Development extension package contains everything we would need and more to use Docker as a remote development environment.

Now let's start generating our development container configuration. Bring up the command palette using Command + Shift + P on Mac and Ctrl + Shift + P on Windows. Type "add dev" and select Add Development Container Configuration Files.

You will see a list of pre-built definitions. Well, go with the Node.js & PostgreSQL since it is already almost everything we need to get RedwoodJS up and running.

You will next be asked which version of Node.js you would like your sandbox to run. Let's stick with the default.

The last window allows you to install additional packages/features, but clicking "OK" in the top right will be good enough.

You should now see a new folder in your project called .devcontainer. Inside it will be 4 files, devconatiner.json, docker-compose.yml, create-db-user.sql and Dockerfile.

Well, ignore create-db-user.sql since it's unimportant for this guide. We also won't be touching the Dockerfile but it's what Docker will use to build a development environment with all bells and whistles we need to run Node.js and more.

Let's open up docker-compose.yml file. We will need to make a minor tweak to it. If you dont know what docker-compose is, it's a tool to tell Docker how to step up containers that will run services we would like. Usually, most web development projects need 2-3 services. One service is your web server, RedwoodJS, for us, and another service would be a database like Postgres.

Out of the box, RedwoodJS will use SQLite, but most of the time, I find myself using Postgres in production, and wouldn't it be cool to emulate production? Using Development Container makes it super easy to use Postgres locally, also. So let's try making our local environment run.

Our new docker-compose.yml already comes pre-configured with an app container we will use to develop inside. Looking closely, you will see this container is based on our Dockerfile image. It also has a db container to run our development Postgres database. We need to add one more thing, a container to run our test database. I have annotated the code below to add a db-test container and corresponding volumes. You might be able to copy and paste the whole thing.

version: '3.8'

services:
  app:
    # Using a Dockerfile is optional, but included for completeness.
    build:
      context: .
      dockerfile: Dockerfile
      args:
        # Update 'VARIANT' to pick an LTS version of Node.js: 18, 16, 14.
        # Append -bullseye or -buster to pin to an OS version.
        # Use -bullseye variants on local arm64/Apple Silicon.
        VARIANT: 16-bullseye

    volumes:
      # This is where VS Code should expect to find your project's source code and the value of "workspaceFolder" in .devcontainer/devcontainer.json
      - ..:/workspace:cached

      # Uncomment the next line to use Docker from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker-compose for details.
      # - /var/run/docker.sock:/var/run/docker.sock

    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity

    # Runs app on the same network as the service container, allows "forwardPorts" in devcontainer.json function.
    network_mode: service:db

    # Use "forwardPorts" in **devcontainer.json** to forward an app port locally.
    # (Adding the "ports" property to this file will not forward from a Codespace.)

    # Uncomment the next line to use a non-root user for all processes - See https://aka.ms/vscode-remote/containers/non-root for details.
    # user: vscode

    # Uncomment the next four lines if you will use a ptrace-based debugger like C++, Go, and Rust.
    # cap_add:
    #   - SYS_PTRACE
    # security_opt:
    #   - seccomp:unconfined

  # You can include other services not opened by VS Code as well
  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_DB: development_db
      POSTGRES_PASSWORD: postgres
  ###### START ADDED SECTION: Create a container to run the test database
  db-test:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-test-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_DB: test_db
      POSTGRES_PASSWORD: postgres
  ###### END ADDED SECTION
volumes:
  postgres-data:
  postgres-test-data:#### ADDED LINE

Now let's open up your .devcontainer.json file. This file is specifically for VSCode. It tells it how to run our development container. More specially, the docker-compose file to use, which container we will use to do development inside, and the extension we would like to install by default. The extension configuration is super lovely, or anyone who works with you will have the same extension automatically installed to make their dev life more effortless. Finally, you can copy and paste the code below into the file.

{
    "name": "RedwoodJS & Postgres (Community)",

    // Update the 'dockerComposeFile' list if you have more compose files or use different names.
    "dockerComposeFile": "docker-compose.yml",

    // The 'service' property is the name of the service for the container that VS Code should
    // use. Update this value and .devcontainer/docker-compose.yml to the real service name.
    "service": "app",

    // The optional 'workspaceFolder' property is the path VS Code should open by default when
    // connected. This is typically a volume mount in .devcontainer/docker-compose.yml
    "workspaceFolder": "/workspace",

    // Configure tool-specific properties.
    "customizations": {
        // Configure properties specific to VS Code.
        "vscode": {
            // Add the IDs of extensions you want installed when the container is created.
            "extensions": [
                "redwoodjs.redwood",
                "dbaeumer.vscode-eslint",
                "ofhumanbondage.react-proptypes-intellisense",
                "mgmcdermott.vscode-language-babel",
                "editorconfig.editorconfig",
                "prisma.prisma",
                "graphql.vscode-graphql"
            ]
        }
    },

    // Uncomment the next line if you want to keep your containers running after VS Code shuts down.
    // "shutdownAction": "none",

    // Uncomment the next line to use 'postCreateCommand' to run commands after the container is created.
    // "postCreateCommand": "uname -a",

    // Comment out to connect as root instead. To add a non-root user, see: https://aka.ms/vscode-remote/containers/non-root.
    "remoteUser": "node"
}

Postgres In Development

The last phase of our changes is going to be with RedwoodJS. Since we're setting up our local development to run Postgres, we need to update some things in our RedwoodJS.

First, let's update our schema.prisma file to run Postgres. Then, update your provider, like in the example below.

datasource db {
  provider = "postgresql" // Switched from MySQL to Postgres for local development
  url      = env("DATABASE_URL")
}

generator client {
  provider      = "prisma-client-js"
  binaryTargets = "native"
}

// Define your own datamodels here and run `yarn redwood prisma migrate dev`
// to create migrations for them and apply to your dev DB.
// TODO: Please remove the following example:
model UserExample {
  id    Int     @id @default(autoincrement())
  email String  @unique
  name  String?
}

Next, we need to open and update our .env file. Since we changed to using Postgres, we need to change DATABASE_URL and TEST_DATABASE_URL to point to our Postgres database containers and use the credentials that are set in the docker-compose.yml. Looking back at the docker-compose.yml file, you will see where we got the username, password, and database name. Another important bit is to set the RWJS_DEV_API_URL variable so routing happens properly from within our container to the GQL server.

# THIS FILE SHOULD NOT BE CHECKED INTO YOUR VERSION CONTROL SYSTEM
#
# Environment variables set here will override those in .env.defaults.
# Any environment variables you need in production you will need to setup with
# your hosting provider. For example in Netlify you can add environment
# variables in Settings > Build & Deploy > environment
#
DATABASE_URL=postgres://postgres:postgres@localhost:5432/development_db
TEST_DATABASE_URL=postgres://postgres:postgres@localhost:5432/test_db
#
# Sets an app-specific secret used to sign and verify your own app's webhooks.
# For example if you schedule a cron job with a signed payload that later will
# then invoke your api-side webhook function you will use this secret to sign and the verify.
# Important: Please change this default to a strong password or other secret
# WEBHOOK_SECRET=THIS_IS_NOT_SECRET_PLEASE_CHANGE
# Set API base URL to be localhost, not 0.0.0.0
RWJS_DEV_API_URL=http://localhost

Start It Up!

Now we're in the home stretch. Let's get your project running in its Development Container. Bring up the command palette using Command + Shift + P on Mac and Ctrl + Shift + P on Windows. Type in "reopen in container" and select "Reopen in Container" option.

This last step will take a little while. It will be downloading the Docker images for your development environment and Postgres, then building it. So sit tight, grab a cup of tea, and get a stretch break-in.

Once done, your VSCode should open up to what looks like a typical project, but with one secret. Your VSCode is connected to your docker container and running inside. The area you will notice the most significant difference will be the terminal.

It says you are inside the workspace folder, and your user is node. If you were to change directories and move up one, you would see completely different folders on your regular computer. You are now in your developer sandbox. Now run yarn redwood dev to start your server and see the magic happen. A bonus to all this work is that it will also work in Github Codespaces.

If there is anything still fuzzy or like me to elaborate further, let me know. I will make sure to do so.

• https://hex.pm/packages/ueberauth_patreon

• https://hex.pm/packages/ueberauth_twitch

Equip with the knowledge of what I needed to do to publish my own package, I thought I would take you on the same journey. Let's get started.

Our first stop is going to be the Hex website. Hex has a great guide for publishing your own package that we will be following.

https://hex.pm/docs/publish

Registering a Hex User

Before we can do anything, we will need to register our own user. When registering a user, you will be prompted for a username, your email, and a password. The email is used to confirm your identity during signup, as well as to contact you in case there is an issue with one of your packages. The email will never be shared with a third party.

$ mix hex.user register

Now it’s time to decide on a name for your package. In this guide I will be creating a new Ueberauth package. If you were to go on http://hex.pm and look at other Ueberauth packages, you notice there is a certain pattern followed. This will make the decision easy for us on what to call our Uberauth package that will implement the Patreon OAuth flow.

uberauth_patreon

Generating Our Project

Now that we know the name of our package let’s generate the project

If you’re curious how to generate a standard mix project, please visit the Elixir site here: https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html#our-first-project

mix new ueberauth_patreon

We should expect to see:

* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating lib
* creating lib/ueberauth_patreon.ex
* creating test
* creating test/test_helper.exs
* creating test/ueberauth_patreon_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd ueberauth_patreon
    mix test

Run "mix help" for more commands.

Let’s change the directory to our new project directory and let’s open the code in your editor of choice. For me I’ll be using Visual Studio Code, this is important later in the guide when I am making a test app.

Next, we will update the mix.exs file to have all the settings we know to start things off. I follow this section of the Hex publishing guide https://hex.pm/docs/publish#adding-metadata-to-code-classinlinemixexscode

def project do
    [
      app: :ueberauth_patreon,
      description: "Ueberauth strategy for Patreon OAuth.",
      version: "1.0.0",
      elixir: "~> 1.13",
      source_url: "https://github.com/talk2MeGooseman/ueberauth_patreon",
      homepage_url: "https://github.com/talk2MeGooseman/ueberauth_patreon",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      package: [
        links: %{"GitHub" => "https://github.com/talk2MeGooseman/ueberauth_patreon"},
        licenses: ["MIT"],
      ]
    ]
  end

I decided to set my package version at 1.0.0 but feel free to start it at 0.0.1 since its your first version ever.

Adding Documentation Generation

If you follow along in the Hex guide, they will suggest using ex_doc and there is no reason not to. It generates some beautiful documentation.

Following the HexDoc’s guide https://hexdocs.pm/ex_doc/readme.html. Let’s add it to our project dependencies in our mix.exs file.

def deps do
  [
        ## Other deps ...
    {:ex_doc, "~> 0.27", only: :dev, runtime: false},
  ]
end

Dont forget to install our new dependency.

mix deps.get

Now let's generate our docs to make sure everything is set up correctly

> mix docs
Generating docs...
View "html" docs at "doc/index.html"
View "epub" docs at "doc/ueberauth_patreon.epub"

If you have the latest NPM installed you can run the following command to see your docs in the browser:

npx serve doc/

This will kick off a little webserver without having to go install anything locally.

Let’s Get Coding

When creating a Ueberauth package, the project needs to be structured in a particular way. If you click on any of the listed strategies Ueberauth has you will get a lot of good examples of what you need to do. You can structure your project any way you like but in the case of Ueberauth it’s best we follow their expected structure.

https://github.com/ueberauth/ueberauth/wiki/List-of-Strategies

Inside of lib, we need to create several files and folders.

mkdir lib/ueberauth
mkdir lib/ueberauth/strategy
touch lib/ueberauth/strategy/patreon.ex
mkdir lib/ueberauth/strategy/patreon
touch lib/ueberauth/strategy/patreon/oauth.ex

Since we're making an Ueberauth strategy we need to make sure to install Ueberauth and OAuth2 packages. The mix.exs file should start looking like this now.

defp deps do
    [
      {:ueberauth, "~> 0.7"},
      {:oauth2, "~> 2.0"},
      {:ex_doc, "~> 0.27", only: :dev, runtime: false}
    ]
end

Now let’s install those new dependencies.

mix deps.get

Now it's time to implement what we want our package to do. I am not going to explain how to implement an Uberauth package in this article, you can read my tutorial Creating Your Own Ueberauth Strategy on how. Instead, we'll just be copying and pasting some code in.

Inside lib/ueberauth/strategy/patreon.ex add the following:

defmodule Ueberauth.Strategy.Patreon do
  use Ueberauth.Strategy,
    oauth2_module: Ueberauth.Strategy.Patreon.OAuth

  alias Ueberauth.Auth.Info
  alias Ueberauth.Auth.Credentials
  alias Ueberauth.Auth.Extra

  @doc """
  Handles the initial redirect to the patreon authentication page.

  To customize the scope (permissions) that are requested by patreon include
  them as part of your url:

      "https://www.patreon.com/oauth2/authorize"
  """
  def handle_request!(conn) do
    scopes = conn.params["scope"] || option(conn, :default_scope)

    params =
      [scope: scopes]
      |> with_state_param(conn)

    module = option(conn, :oauth2_module)
    redirect!(conn, apply(module, :authorize_url!, [params]))
  end

  @doc """
  Handles the callback from Patreon.

  When there is a failure from Patreon the failure is included in the
  `ueberauth_failure` struct. Otherwise the information returned from Patreon is
  returned in the `Ueberauth.Auth` struct.
  """
  def handle_callback!(%Plug.Conn{params: %{"code" => code}} = conn) do
    module = option(conn, :oauth2_module)
    token = apply(module, :get_token!, [[code: code]])

    if token.access_token == nil do
      set_errors!(conn, [
        error(token.other_params["error"], token.other_params["error_description"])
      ])
    else
      fetch_user(conn, token)
    end
  end

  @doc false
  def handle_callback!(conn) do
    set_errors!(conn, [error("missing_code", "No code received")])
  end

  @doc """
  Cleans up the private area of the connection used for passing the raw Notion
  response around during the callback.
  """
  def handle_cleanup!(conn) do
    conn
    |> put_private(:patreon_token, nil)
    |> put_private(:patreon_user, nil)
  end

  @doc """
  Fetches the uid field from the Twitch response. This defaults to the option `uid_field` which in-turn defaults to `id`
  """
  def uid(conn) do
    %{"data" => user} = conn.private.patreon_user
    user["id"]
  end

  @doc """
  Includes the credentials from the Patreon response.
  """
  def credentials(conn) do
    token = conn.private.patreon_token

    %Credentials{
      token: token.access_token,
      token_type: token.token_type,
      refresh_token: token.refresh_token,
      expires_at: token.expires_at
    }
  end

  @doc """
  Fetches the fields to populate the info section of the `Ueberauth.Auth`
  struct.
  """
  def info(conn) do
    %{ "data" => %{
      "attributes" => %{
        "full_name" => full_name,
        "first_name" => first_name,
        "last_name" => last_name,
        "about" => about,
        "image_url" => image_url,
        "url" => url,
        "email" => email
      }
    }} = conn.private.patreon_user

    %Info{
      email: email,
      name: full_name,
      first_name: first_name,
      last_name: last_name,
      description: about,
      image: image_url,
      urls: %{
        profile: url
      }
    }
  end

  @doc """
  Stores the raw information (including the token) obtained from the Patreon
  callback.
  """
  def extra(conn) do
    %Extra{
      raw_info: conn.private.patreon_user
    }
  end

  defp fetch_user(conn, token) do
    conn = put_private(conn, :patreon_token, token)

    case Ueberauth.Strategy.Patreon.OAuth.get(
           token.access_token,
           "https://www.patreon.com/api/oauth2/v2/identity?fields%5Buser%5D=full_name,email,first_name,last_name,about,image_url,url"
         ) do
      {:ok, %OAuth2.Response{status_code: 401, body: _body}} ->
        set_errors!(conn, [error("token", "unauthorized")])

      {:ok, %OAuth2.Response{status_code: status_code, body: user}}
      when status_code in 200..399 ->
        put_private(conn, :patreon_user, user)

      {:error, %OAuth2.Error{reason: reason}} ->
        set_errors!(conn, [error("OAuth2", reason)])

      {:error, %OAuth2.Response{body: %{"message" => reason}}} ->
        set_errors!(conn, [error("OAuth2", reason)])

      {:error, _} ->
        set_errors!(conn, [error("OAuth2", "uknown error")])
    end
  end

  defp option(conn, key) do
    Keyword.get(options(conn), key, Keyword.get(default_options(), key))
  end
end

Inside lib/ueberauth/strategy/patreon/oauth.ex paste the following code:

defmodule Ueberauth.Strategy.Patreon.OAuth do
  @moduledoc """
  An implementation of OAuth2 for Patreon.

  To add your `:client_id` and `:client_secret` include these values in your
  configuration:

      config :ueberauth, Ueberauth.Strategy.Patreon.OAuth,
        client_id: System.get_env("PATREON_CLIENT_ID"),
        client_secret: System.get_env("PATREON_CLIENT_SECRET")

  """
  use OAuth2.Strategy

  @defaults [
    strategy: __MODULE__,
    site: "https://www.patreon.com/",
    authorize_url: "https://www.patreon.com/oauth2/authorize",
    token_url: "https://www.patreon.com/api/oauth2/token",
    token_method: :post
  ]

  @doc """
  Construct a client for requests to Patreon.

  Optionally include any OAuth2 options here to be merged with the defaults:

      Ueberauth.Strategy.Patreon.OAuth.client(
        redirect_uri: "http://localhost:4000/auth/patreon/callback"
      )

  This will be setup automatically for you in `Ueberauth.Strategy.Patreon`.

  These options are only useful for usage outside the normal callback phase of
  Ueberauth.
  """
  def client(opts \\ []) do
    config =
      :ueberauth
      |> Application.fetch_env!(Ueberauth.Strategy.Patreon.OAuth)
      |> check_credential(:client_id)
      |> check_credential(:client_secret)
      |> check_credential(:redirect_uri)

    client_opts =
      @defaults
      |> Keyword.merge(config)
      |> Keyword.merge(opts)

    json_library = Ueberauth.json_library()

    OAuth2.Client.new(client_opts)
    |> OAuth2.Client.put_serializer("application/json", json_library)
    |> OAuth2.Client.put_serializer("application/vnd.api+json", json_library)
  end

  @doc """
  Provides the authorize url for the request phase of Ueberauth.

  No need to call this usually.
  """
  def authorize_url!(params \\ [], opts \\ []) do
    opts
    |> client
    |> OAuth2.Client.authorize_url!(params)
  end

  def get(token, url, headers \\ [], opts \\ []) do
    client()
    |> put_header("authorization", "Bearer " <> token)
    |> put_header("accept", "application/json")
    |> OAuth2.Client.get(url, headers, opts)
  end

  def get_token!(params \\ [], options \\ []) do
    headers = Keyword.get(options, :headers, [])
    options = Keyword.get(options, :options, [])

    client_options = Keyword.get(options, :client_options, [])

    client = OAuth2.Client.get_token!(client(client_options), params, headers, options)

    client.token
  end

  # Strategy Callbacks
  def authorize_url(client, params) do
    OAuth2.Strategy.AuthCode.authorize_url(client, params)
  end

  def get_token(client, params, headers) do
    client =
      client
      |> put_param("grant_type", "authorization_code")
      |> put_header("Accept", "application/json")

    OAuth2.Strategy.AuthCode.get_token(client, params, headers)
  end

  defp check_credential(config, key) do
    check_config_key_exists(config, key)

    case Keyword.get(config, key) do
      value when is_binary(value) ->
        config

      {:system, env_key} ->
        case System.get_env(env_key) do
          nil ->
            raise "#{inspect(env_key)} missing from environment, expected in config :ueberauth, Ueberauth.Strategy.Patreon"

          value ->
            Keyword.put(config, key, value)
        end
    end
  end

  defp check_config_key_exists(config, key) when is_list(config) do
    unless Keyword.has_key?(config, key) do
      raise "#{inspect(key)} missing from config :ueberauth, Ueberauth.Strategy.Patreon"
    end

    config
  end

  defp check_config_key_exists(_, _) do
    raise "Config :ueberauth, Ueberauth.Strategy.Patreon is not a keyword list, as expected"
  end
end

It’s always a great idea to add doc headers to all your public functions you expect your package users to be using. This will help them immensely in figuring out what each function does and seeing examples of input and output. For this package, a user won't be using it directly but instead following specific setup steps, so our documentation reflects that.

Generate our documentation to see how things look now.

mix docs

And spin up a server to see how things.

npx serve docs/

Bonus: Add the README

This is more of a personal preference than anything else. I'm not sure if you noticed when browsing the Hex Docs for various packages. But sometimes you will come across a project that has pretty nice docs but the setup steps will be missing. If you actually visit the packages repository, you will find a great README with all the information on how to setup up the project.

GAHHHH I also need those steps in those docs! Well, you easily can tell ex_doc to include the README. Let’s update our project settings to do just that.

In our mix.exs file add the docs field, it should look something like this now.

def project do
    [
      app: :ueberauth_patreon,
      description: "Ueberauth strategy for Patreon OAuth.",
      version: "1.0.0",
      elixir: "~> 1.13",
      source_url: "https://github.com/talk2MeGooseman/ueberauth_patreon",
      homepage_url: "https://github.com/talk2MeGooseman/ueberauth_patreon",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      package: [
        links: %{"GitHub" => "https://github.com/talk2MeGooseman/ueberauth_patreon"},
        licenses: ["MIT"],
      ],
      docs: [ #### New docs field
        extras: ["README.md"] ### This is going to include the readme in the docs
      ]
    ]
  end

Now let’s go to our README and update it with the project setup steps.

# Überauth Patreon

[![Hex Version](https://cdn.hashnode.com/res/hashnode/image/upload/v1679633621626/fe2dc713-2811-4631-8d98-6abd29a2d38b.svg)](https://hex.pm/packages/ueberauth_patreon)

> Patreon OAuth2 strategy for Überauth.

## Installation

1. Setup your application in Patreon Development Dashboard https://www.patreon.com/portal/registration/register-clients

1. Add `:ueberauth_patreon` to your list of dependencies in `mix.exs`:

    ```elixir
    def deps do
      [{:ueberauth_patreon, "~> 1.0"}]
    end

1. Add Patreon to your Überauth configuration:

    config :ueberauth, Ueberauth,
      providers: [
        patreon: {Ueberauth.Strategy.Patreon, [default_scope: "identity[email] identity"]},
      ]
   

2. Update your provider configuration:

   config :ueberauth, Ueberauth.Strategy.Patreon.OAuth,
     client_id: System.get_env("PATREON_CLIENT_ID"),
     client_secret: System.get_env("PATREON_CLIENT_SECRET"),
     redirect_uri: System.get_env("PATREON_REDIRECT_URI")
   

3. Include the Überauth plug in your router pipeline:

   defmodule TestPatreonWeb.Router do
     use TestPatreonWeb, :router
   
     pipeline :browser do
       plug Ueberauth
       ...
      end
   end
   

4. Add the request and callback routes:

   scope "/auth", TestPatreonWeb do
     pipe_through :browser
   
     get "/:provider", AuthController, :request
     get "/:provider/callback", AuthController, :callback
   end
   

5. Create a new controller or use an existing controller that implements callbacks to deal with Ueberauth.Auth and Ueberauth.Failure responses from Patreon.

      defmodule TestPatreonWeb.AuthController do
        use TestPatreonWeb, :controller
   
        def callback(%{assigns: %{ueberauth_failure: _fails}} = conn, _params) do
          conn
          |> put_flash(:error, "Failed to authenticate.")
          |> redirect(to: "/")
        end
   
        def callback(%{assigns: %{ueberauth_auth: auth}} = conn, _params) do
          case UserFromAuth.find_or_create(auth) do
            {:ok, user} ->
              conn
              |> put_flash(:info, "Successfully authenticated.")
              |> put_session(:current_user, user)
              |> configure_session(renew: true)
              |> redirect(to: "/")
   
            {:error, reason} ->
              conn
              |> put_flash(:error, reason)
              |> redirect(to: "/")
          end
        end
      end
   

Calling

Once your setup, you can initiate auth using the following URL, unless you changed the routes from the guide:

/auth/patreon

Documentation

The docs can be found at ueberauth_patreon on Hex Docs.

Generate our documentation to see how things look now.

```bash
mix docs

And spin up a server to see how things.

npx serve docs/

Our new package is code complete but we don't know if it actually works.

Testing Our Package In A Project

I think I write half-decent code, but not enough to trust that it just works so I want to try testing it in an actual Phoenix project. Let’s spin one up to test it LIVE.

mix phx.new test_patreon

Short Detour: Using VS Code Run Our Project

We will need a database for our project, the easiest way I like to do this is using Visual Studio Code’s Remote - Containers feature. Using Docker and an existing template, I can spin up a dev environment that all the fixings I need to work on a Phoenix project.

1. Start VS Code, run the Remote-Containers: Open Folder in Container... command from the Command Palette (F1) or quick actions Status bar item, and select the project folder you would like to set up the container for.

2. Now select Show All Definitions... > Elixir, Phoenix, Node.js & PostgresSQL (Community)

3. After picking the starting point for your container, VS Code will add the dev container configuration files to your project (.devcontainer/devcontainer.json).

4. The VS Code window will reload and start building the dev container. A progress notification provides status updates. You only have to build a dev container the first time you open it; opening the folder after the first successful build will be much quicker.

5. After the build completes, VS Code will automatically connect to the container.

Now your project is all set up with everything you need to run the project. You will most likely need to go into the docker-compose.yml file and update the database name to the name your project will be using.

Resuming: Testing Our Package In A Project

Let make sure our project setup works.

mix setup

In order to test our project, we don’t need to worry about publishing it to Hex. We can install it locally or install it through git. I have already pushed my project to Github so I want to show you how you can install any package hosted on Git or Github really easily.

Add the following to your deps. Your name or URL would be different if your project.

{:ueberauth_patreon, github: "talk2MeGooseman/ueberauth_patreon"}

As a personal rule of thumb, make sure to go through the setup guide you have for your package VERBATIM. This ensures you didn't miss any steps as part of the setup guide, because were the author sometimes we can accidentally gloss over an important.

Now install your package and test it out.

mix deps.get

Here is to hoping your project just works during testing it!

UH OH, My Package Has A Bug!

When the inevitable and your find a bug in your code there are a couple of things you can do.

1. Make updates to your package project code and push the changes to Git. If you do this, in your test project make sure to run the following command to install the latest updates.

mix deps.update ueberauth_patreon

1. Modify the package code directly inside your test project. All your installed packages can be found inside your deps/ folder and any changes you make to them can be recompiled so you can see how they affect your project. This can save you the trouble of having to go back to your package project, make code changes, and then push the code. Just run the following command to recompile your package.

mix deps.compile ueberauth_patreon

Make sure you copy the fixes you make back to your package project and commit them!

Submitting The Package

Now a package is complete and is ready to publish. Hex again does a great job detailing the steps you need to go through here https://hex.pm/docs/publish#submitting-the-package

Just run the command and confirm when you verify everything looks good.

> mix hex.publish
Building ueberauth_patreon 1.0.0
  Dependencies:
    ueberauth ~> 0.7 (app: ueberauth)
    oauth2 ~> 2.0 (app: oauth2)
  App: ueberauth_patreon
  Name: ueberauth_patreon
  Files:
    lib
    lib/ueberauth
    lib/ueberauth/strategy
    lib/ueberauth/strategy/patreon
    lib/ueberauth/strategy/patreon/oauth.ex
    lib/ueberauth/strategy/patreon.ex
    lib/ueberauth_patreon.ex
    .formatter.exs
    mix.exs
    README.md
  Version: 1.0.0
  Build tools: mix
  Description: Ueberauth strategy for Patreon OAuth.
  Licenses: MIT
  Links: 
    GitHub: https://github.com/talk2MeGooseman/ueberauth_patreon
  Elixir: ~> 1.13
Before publishing, please read the Code of Conduct: https://hex.pm/policies/codeofconduct

Publishing package to public repository hexpm.

Oh No I Messed Up My Docs!

Free not, for Hex has a way to directly publish doc updates. No need to make a new release just to update docs. Once you have made your docs updates just run the following command and you are good.

mix hex.publish docs

Transcript of Video Below:

How's it going, folks. And thanks for checking out my video on lessons learned deploying elixir. So in this video, I want to talk about a situation that occurred shortly after I deployed my first elixir Phoenix application. So the site I deployed is called stream closed captioner. And it is a solution to add closed captions to a Twitch stream, to zoom meeting, OBS via OBS web sockets. This Phoenix application relies on web sockets to send messages from the client over to the backend, and then that gets sent over to Twitch.

It is a fairly straightforward pipeline, nothing too sophisticated. There isn't really much in the way of CPU-intensive calculations or anything like that. All it does is a query for user information regarding what it should do in terms of authentication, user settings, additional filtering to happen for the text to speech.

So I want to talk about a situation that happened shortly after deployment for this elixir application. For my hosting solution, I am using Gigalixir, and Gigalixir comes with a very nice and simple interface to see memory usage and CPU cores. This is the current implementation I have, and it's fairly straightforward. It is just two replicas and using one gigabyte of memory. After the deployment, I didn't see a very consistent graph. Instead, I saw something like this. So this is a screenshot of what was happening roughly two weeks ago; right after deployment, I had deployed the application at 11:00 PM, and I saw huge spikes in CPU usage.

This is at 11:00 PM, so there's not a lot of traffic. Again, there's not anything computationally heavy with the application. And I also saw huge spikes in memory. So much so that I saw out-of-memory errors, and that's not good. And Elixir is supposed to be fast, memory efficient.

This is my first elixir application. So I was like, what the heck is going on? This is fine, it's running, but it's not really fine. I'm in the middle of a fire. Things are running, but I'm hitting out of memory errors. I don't know what's going on. Locally, my Elixir application runs fine. It doesn't look like there's anything crazy going on, but there's obviously something very wrong. Something that doesn't fit with exactly what a Phoenix Elixir application is supposed to be. I'm not doing anything. Memory, CPU intensive. I'm not doing anything crazy with it. It is just a straightforward web socket application that's sending messages and posting API requests, nothing crazy. So I needed to dig in and figure out what the heck is going on because code-wise, everything looked straightforward. Luckily, I had instrumented my application to use new Relic.

So here's the new Relic dashboard, and New Relic is awesome because it provides a lot of great information about your application, the web requests, the throughput. One of the really awesome things about it also has, for elixir applications, a beam profiler so it can connect to the beam process and give you information about what is going on in the application. So I went into here, and I checked things out. I specifically looked at the processes, and nothing out of the ordinary was happening. So I went to the memory tab. So right now, this is the current application running in production.

As of today, today is June 6th. This is an example of fairly consistent memory usage for the application. The table sizes, kind of increasing, decreasing as traffic happens and as garbage collection occurs, but there is a pretty consistent, memory usage across the board. Now at the time where I was facing, out of memory errors, CPU spikes.

This is what I was saying. Unfortunately, with new Relic, the timeline to go back and look at memory and processes doesn't go back too far. It only goes about two weeks, but luckily I took some screenshots of what I saw, and I saw huge memory spikes. Ever since the deployment, you could see like here there are memory spikes here. There's a huge memory spike here, and it just wasn't making any sense what the heck is going on out of the total memory. So I kept on digging in, and I took a look at the process memory, and you can see what the process memory I was. I saw spikes again, huge, huge memory spikes there. But the cool thing about the process memory is that these lines represent the processes running in elixir, and everything's a process in elixir.

So I could see right here that boom notifier. This process is the thing that seems to be taking up lots of memory. It's always taking up memory. Regardless of whether I restarted the application, redeployed any changes it's, it's taking up memory. It's always spiking up pretty high. And I was like, Hmm.

Okay. Well, I know what boom notifier is, it's a notification package that I wanted to use to get emailed exceptions that happen in the application because it's a similar plugin that I was using in a Ruby on rails application I had been using. So I just wanted to receive emails whenever an exception happens, so I can quickly track those.

Here's the package itself, boom, notifier. And I followed all the guides. It was fairly straightforward to use and configure. But as we saw on this chart right here, it looks like boom notifier is showing some memory peaks. So okay, well, I don't need boom notifier because new Relic offers error tracking in case there are 500 errors, exceptions being thrown.

So I don't need boom notifier. So let me try and get rid of boom notifier. And see what happens. I had taken it out of the mix.exs, committed the changes, and deployed them. And so what happened. Now, if we go back to this graph over here and we see these spikes, we see a huge drop off right here at the end, this right here is after boom notifier was removed and the application was deployed. So I'm not trying to knock boom notifier or anything. It's a solid plugin for your application, but in my situation, it was causing some huge CPU and memory usage problems. And you can see right here that after I removed it, it went down significantly.

And now I was below 1.0 CPU usage. And also memory dropped down to a very, very consistent level.

Which was awesome. So after removing this package, I saw a significant gain and performance. And if we go back here, we had the spikes here and after. The deployment, removing boom notifier. You see a very consistent level of memory here and the same thing over here, the process memory drops down by a lot.

So what is the lesson that I learned? Right out of the gate with Elixir and packages. It's to be careful, be very careful of what you put into your application because if you don't need it, don't use it.

So after that, now I have a very consistent graph of memory usage across the board and my application no longer runs out of memory and no longer has drastic CPU spikes. It is very consistent, very efficient, and it's very much what an elixir application should be. If there's anything I want you to take away from this video, it's that just be careful with your packages and use the tools necessary to keep track of your application, to see what the performance is, what the memory usage is. So you can identify things that could be hindrances, could be bottlenecks for your application in production. I got lucky that I had set up New Relic ahead of time.

I know there's another tooling that you can use for elixir, but New Relic was the easiest thing for me since I am fairly new to Elixir itself and running it in production.

So hopefully you learn something new from this, and I will see you in the next video. Thanks, everyone.

For a while now, I have been using Docker for local development on my Ruby on Rails applications. This has worked out great and I've spent plenty of time tweaking Docker to suit my needs, as things have changed. I recently started a new initiative to use Docker in a production environment, not just for local development. Unfortunately, I hit a little hiccup.

Setting Up Docker: The Problem

If you are like me, the first time you configured your Dockerfile for Ruby on Rails, you found a guide to set up your Dockerfile. It was great for setting up a local development environment. It probably had you set up a Dockerfile, create a docker-compose.yml file, and set up a build and run process. Here is a very abbreviated example of my Dockerfile

FROM ruby:2.5.3
###############################################################################
# Base Software Install
###############################################################################
RUN curl -sL https://deb.nodesource.com/setup_$RAILSDOCK_NODE_VERSION.x | bash -
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
 echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
.
.
.
###############################################################################
# Ruby, Rubygems, and Bundler Defaults
###############################################################################
ENV LANG C.UTF-8
.
.
.
ENV RAILS_ENV production
###############################################################################
# Final Touches
###############################################################################
.
.
.
RUN bundle config build.nokogiri --use-system-libraries
RUN bundle check || bundle install
# Copy for package.json and yarn.lock to do install to save layer size
COPY package.json yarn.lock ./
RUN yarn install --check-files --production=true
# Finally copy over rest of app
COPY . /app
# Precompile assets for production
RUN bundle exec rake assets:precompile
.
.
.

If you just do the standard tutorials, everything works great in development. But, most of these tutorials don't cover the .dockerignore file. This file is very important and not having one becomes a problem when Docker runs the following line in the above Dockerfile:

# Finally copy over rest of app
COPY . /app

Locking Down Your Docker Image For Production or Sharing

By default, when you run a Docker build and copy your directory into the image, it will grab every single file in your directory: the node_modules, those weird .DS_Store files, and even your .env files that could contain sensitive information. Why is this a problem? If you happen to make your Docker image publicly available after it's built, people can look inside and see those extra files that were included inside. Most Dockerize your Rails app guides don't cover. The solution is to use a .dockerignore file. It functions exactly like a .gitignore file. Put the files and folder paths that you don't want committed to your Docker image. Here is an example of my .dockerignore file.

**/.classpath
**/.dockerignore
**/.env
**/.git
**/.gitignore
**/.project
**/.settings
**/.toolstarget
**/.vs
**/.vscode
**/*.*proj.user
**/*.dbmdl
**/*.jfm
**/azds.yaml
**/charts
**/docker-compose*
**/Dockerfile*
**/node_modules
**/npm-debug.log
**/obj
**/secrets.dev.yaml
**/values.dev.yaml
README.md
.DS_Store
.bin
.git
.gitignore
.bundleignore
.bundle
.byebug_history
.rspec
tmp
log
test
config/deploy
config/master.key
public/packs
public/packs-test
node_modules
yarn-error.log
coverage/

This keeps a lot of development cruft out of my Docker image. Reviewing it, I could probably update it again 😁. Once you add your .dockerignore file to your project, try to build things again. Now, your Docker container should be leaner (no more extra files you don't need in there) and more secure!