Landing on Elixir: Processing Immutable Data

The pain of processing immutable data

The other day I saw a block of code whose author had struggled in processing the data. Here’s a simplified version of it:

defp transform_store(store) do
  retailers = store["retailers"]
  logo_url = retailers["logoUrl"]

  retailers =
    if logo_url != nil do
      retailers = Map.delete(retailers, "logoUrl")
      logo_url = @prefix <> logo_url
      Map.put(retailers, "logoUrl", logo_url)
    else
      retailers
    end

  retailer_id = retailers["id"]
  company = retailers["company"]
  company_map_id = company["id"]
  company = Map.delete(company, "id")
  company = Map.put(company, "_id", company_map_id)
  retailers = Map.delete(retailers, "id")
  retailers = Map.delete(retailers, "company")
  retailers = Map.put(retailers, "_id", retailer_id)
  retailers = Map.put(retailers, "company", company)

  store_map_id = store["id"]
  store = Map.delete(store, "id")
  store = Map.delete(store, "retailers")
  store = Map.put(store, "retailer", retailers)
  store = Map.put(store, "_id", store_map_id)

  store
end

I constrained to figure out that what it does is to transform some data with the following shape:

%{
  "point" => [_latitude, _longitude],
  "id" => "store-id",
  "retailers" => %{
    "id" => "retailer-id",
    "logoUrl" => "/path/to/logo",
    "company" => %{
      "id" => "company-id"
    }
  }
}

into:

%{
  "_id" => "store-id",

  # note that `retailers` has been renamed to
  # `retailer` (singular form)
  "retailer" => %{
    "_id" => "retailer-id",
    "logoUrl" => @prefix <> "/path/to/logo",
    "company" => %{
      "_id" => "company-id"
    }
   }
}

I felt the struggle in writing the code because the author was not familiar with APIs provided by Elixir.

But indeed, processing immutable data in Elixir can be more fun. In this article, we'll discuss some methods to process data in Elixir, and see how this code can be written into maintainable and reusable code.

To warm-up, let's discuss some basic concepts of processing immutable data.

Immutable Data

Immutable data is not a new concept. It is a fundament of functional programming. But this may not be obvious at first, and we may struggle in dealing with data, especially in a deeply nested shape.

Technically, we can not update the data. What we can do is using a function to transform it into a new piece of data, leaving the original data unchanged, as the following depicts:

The original data (O) is still available after processed, and the new output data (O') is different. They can share some common parts for better performance, but they are two objects.

The concept is simple, but we need to get used to it because it leads to different processing patterns than mutable data.

Click on the titles to expand them.

data = %{"id" => "ABCDEF"}

iex> downcase_id.(data)
%{"id" => "abcdef"}

downcase_id = fn %{"id" => id} = data ->
  %{data | "id" => String.downcase(id)}
end

downcase_id = fn data ->
  Map.update!(data, "id", &String.downcase/1)
end

# from:
%{"_id" => "example"}

# to:
%{"id" => "example"}

fn data ->
  # first we remove the old key ("_id"), and save its value
  {id_value, temp_data} = Map.pop(data, "_id")

  # optionally, restructure we may compute a new value based on the
  # original value:
  #
  # id_value = process_id_value(id_value)

  # then we put back the id value under a new key ("id")
  Map.put(temp_data, "id", id_value)
end

%{
  a: %{
    b: %{
      c: %{
        "id" => "EXAMPLE" # <- change it to "example"
      }
    }
  }
}

fn data ->
  Map.update!(data, :a, fn data_a ->
    Map.update!(data_a, :b, fn data_b ->
      Map.update!(data_b, :c, fn data_c ->
        Map.update!(data_c, "id", &String.downcase/1)
      end)
    end)
  end)
end)

fn data ->
  update_in(data.a.b.c["id"], &String.downcase/1)
end

# before
["a", "b", "c", "d"]

# after
["a", "B", "c", "d"]

fn data ->
  List.update_at(data, 1, &String.upcase/1)
end

# orignal data
%{
  "data" => %{
-   "logo" => "http://example.com/logo.png",
+   "logo" => "http://image.example.com/logo.png",
    "articles" => [
      %{
        "title" => "Article 1",
        "image" => %{
-         "url" => "http://example.com/1.png"
+         "url" => "http://image.example.com/1.png"
        }
      },
      %{
        "title" => "Article 2",
        "image" => %{
-         "url" => "http://example.com/2.png"
+         "url" => "http://image.example.com/2.png"
        }
      }
    ]
  }
}

def process(map) when is_map(map),
  do: map
      |> Enum.map(fn {key, value} ->
        {key, process(value)}
      end)
      |> Enum.into(%{})

def process(map) when is_map(map) do
  for {key, value} <- map, into: %{} do
    {key, process(value)}
  end
end

def process(map) when is_map(map),
  do: map
      |> Map.new(fn {key, value} ->
        {key, process(value)}
      end)

def process(list) when is_list(list),
  do: Enum.map(list, &process/1)

def process(url) when is_binary(url),
  do: String.replace(
    url,
    "http://example.com/",
    "http://image.example.com/"
  )

def process(other), do: other

def process(map) when is_map(map) do
  for {key, value} <- map, into: %{} do
    {key, process(value)}
  end
end

def process(list) when is_list(list),
  do: Enum.map(list, &process/1)

def process(url) when is_binary(url),
  do:
    String.replace(
      url,
      "http://example.com/",
      "http://image.example.com/"
    )

def process(other), do: other

The refactored code

Let's look back at the code mentioned at the beginning. The code smells bad because it uses Map.delete/2 and Map.put/3 only which are not ideal here. It can be refactored as:

def transform_store(store) do
  store
  # Let's separate the update on retailer into another small function.
  |> Map.update!("retailers", &transform_retailer/1)

  # Since replacing key name is so common, it is now a new function.
  |> replace_key_in(~w[id], "_id")

  # We reuse the function to change map key.
  |> replace_key_in(~w[retailers], "retailer")
end

defp transform_retailer(retailer) do
  retailer
  # Here we use `Map.update/4` to update the logo url.
  # The result is slightly different than the original
  # version but should be OK.
  |> Map.update("logoUrl", nil, &(@prefix <> &1))

  # Again, we use `replace_key_in/3` to change key name.
  |> replace_key_in(~w[id], "_id")
  |> replace_key_in(~w[company id], "_id")
end

And here's the definition of replace_key_in/3:

defp replace_key_in(data, [old_name], new_name),
  do: replace_key(data, old_name, new_name)

defp replace_key_in(data, [_ | _] = path, new_name) do
  {path, [old_name]} = Enum.split(path, -1)

  update_in(
    data,
    path,
    &replace_key(&1, old_name, new_name)
  )
end

defp replace_key(data, old_name, new_name) do
  {value, rest} = Map.pop(data, old_name)
  Map.put(rest, new_name, value)
end

Let's quickly recap what we have done to improve it.

1. Since replacing a key in a map is so frequent here, we write a helper function replace_key_in/3 to keep the domain function clear to read. Furthermore, this helper function can be extracted to another place which can be reused by other modules or projects.
2. We use Kernel.update_in/3 to update the deeply nested map.
3. Several Map APIs (Map.update!/3, Map.pop/2, and Map.put/3) are employed to serve different purposes.

I believe the refactored version is more readable, maintainable, and reusable.

Conclusion

We have discussed several patterns to update some parts of the data. I hope they are helpful to you!

Further reading

• Map module documentation
• List module documentation