In my last few posts, I wrote about the persistent ZINGG_ID and incremental flows. Quick recap for those who missed them. As data keeps changing in a growing business - new customers, updated addresses, records arriving from new channels - the identity graph has to keep up. The incremental flow in Zingg Enterprise handles exactly this. New and updated records get matched against existing clusters, clusters merge and split automatically, and the ZINGG_ID travels with the entity through all of it. The identity graph stays consistent even as the data underneath it keeps moving.

That was a big piece of work to build and ship. But as we rolled it out to more customers, we started hearing about a different scenario. One we had not fully planned for.

Some of our customers had been running Zingg for a while. Their identity graphs were in good shape. Incremental flows were humming. And then, their data got better.

Phone numbers that were unreliable at the start of the project - maybe coming from a poorly integrated source system, or just sparsely captured - were now being collected consistently. Email fields that were mostly empty were now populated. A new column from a recently onboarded data source was throwing up a great matching signal. The data engineering team had done some serious cleanup work on one of the source systems.

In other words, the matching could now be sharper than it was when the graph was first built.

And yet, teams were not acting on it. We started hearing the same thing again and again. Customers who loved their ZINGG_IDs - and were relying on them deeply - were finding that the very thing they valued was also holding them back. One customer had significant architecture changes and needed to run a full refresh, but was worried about losing their ZINGG_IDs. Others wanted to add a new matching signal but held off. Some wanted to move to a different data platform entirely, but the ZINGG_ID dependency made the migration feel too risky.

The Zingg identifier had become load-bearing infrastructure. Which we loved to see - it meant the graph was genuinely embedded in how the business operated. But it also meant that any change to the graph felt dangerous. Customers were not able to move from one model to another, capture more signals, or even change their data platform because of Zingg. That was not the position we wanted our customers to be in.

So, what do you do when you want a full refresh - run matching again on the full dataset, this time with the richer fields, the better signal, the cleaner data? But your downstream systems are wired to the existing ZINGG_IDs.

Reporting tables, Customer 360 dashboards, ML feature stores, operational workflows at the call centre - all of them reference the ZINGG_IDs from the previous run. If you blow away the graph and rebuild it, you get better clusters but you lose the identifier continuity. Every downstream system breaks or needs a migration. That is a huge cost, and it tends to make teams postpone the refresh indefinitely. The identity graph gets frozen at the quality level it was when it was first built, because improving it feels too disruptive.

This is the gap that Reassign closes.

After you run a full refresh with your updated fields and retrained model, reassign maps the new cluster assignments back to the existing ZINGG_IDs. It carries the established identifiers forward wherever the entity continuity holds. Downstream systems keep their reference point. The identity graph gets the benefit of the improved matching signal. The refresh is no longer a disruptive operation.

Here is the even more interesting bit - what taking the persistent ZINGG_ID one step further really means. Persistence, as we built it earlier, means the identifier travels with the entity through routine changes - new records, updated records, incremental loads. Reassign extends that guarantee to cover a deliberate, intentional rebuild of the graph itself. The identifier now survives not just the churn of normal operations, but also the data team’s decision to go back and do it better.

We think of this as data maturity support built into the identity graph. Because in practice, the signals get better over time. Source systems get cleaned up. New integrations bring in new attributes. The identity graph should be able to evolve with that maturity - not fight against it.

If you have a full refresh sitting in your backlog because the downstream impact felt too risky, we would love to hear from you. Hit reply and tell us what triggered it - what improved signal are you sitting on that you have not yet been able to incorporate?

In this tutorial, we’ll explore what seasonality is, why it matters, and how to build a TradingView-style seasonality heatmap in Python using the OpenAlgo SDK and Plotly.

What is Seasonality?

Seasonality refers to the tendency of financial markets to exhibit recurring patterns at specific times of the year. These patterns emerge from a mix of structural, behavioral, and institutional factors that repeat on a calendar cycle.

For example, you may have heard of the classic Wall Street adage:

“Sell in May and go away.”

This isn’t just folklore. Data across decades and geographies shows that the May–October period tends to underperform the November–April period. Seasonality captures exactly these kinds of tendencies — not as guarantees, but as statistical tilts that can inform better decision-making.

Why Does Seasonality Matter?

1. It Reveals Hidden Probabilities

When you look at a stock’s monthly return heatmap over 10+ years, you start to see structure where there appeared to be randomness. A month that has been positive 80% of the time across a decade isn’t a coincidence — it’s a signal worth paying attention to.

2. It Improves Trade Timing

Imagine you have a bullish thesis on a stock. Seasonality data can help you decide when to enter. If the stock historically rallies in October and dips in September, you might time your entry at the September dip rather than chasing the October move.

3. It Adds a Layer of Confluence

No single tool should drive your trading decisions. But when your technical setup, fundamental thesis, and seasonal tendency all align — that’s confluence. Seasonality acts as a confirmation layer that increases conviction.

4. It Helps with Risk Management

Months with historically high standard deviation warn you to expect volatility. Months with low positive percentages signal caution. This information helps you size positions appropriately and set realistic expectations.

5. It Works Across Asset Classes

Seasonality isn’t limited to equities. Commodities (like crude oil and gold), currencies, and even crypto markets exhibit seasonal patterns driven by harvest cycles, fiscal year-end flows, holiday spending, and institutional rebalancing.

Key Metrics in Seasonal Analysis

A proper seasonality study goes beyond just looking at average returns. Here are the three metrics that matter:

Average Return (Avgs): The mean monthly return across all years. This tells you the directional bias — whether a month tends to be positive or negative on average.

Standard Deviation (StDev): The dispersion of returns around the average. A high StDev means the month is volatile and unpredictable, even if the average looks attractive. A low StDev with a positive average is the sweet spot.

Percent Positive (Pos%): The percentage of years in which the month delivered a positive return. This is arguably the most actionable metric. An average of +5% means little if the month was positive only 40% of the time — it could be skewed by one outlier year. But a Pos% of 80%+ with even a modest average suggests a reliable seasonal tendency.

Building a Seasonality Heatmap in Python

Let’s build a seasonality heatmap that mirrors TradingView’s built-in Seasonality indicator — complete with the same dark theme, color-coded cells, and summary statistics.

Prerequisites

Install the required packages:

pip install openalgo plotly pandas numpy

Make sure your OpenAlgo application is running locally at http://127.0.0.1:5000 and you have a valid API key.

The Complete Code

import pandas as pd
import numpy as np
import plotly.graph_objects as go
from openalgo import api

# ─── Configuration ───────────────────────────────────────────────
API_KEY = "your_api_key_here"
HOST = "http://127.0.0.1:5000"
SYMBOL = "ICICIBANK"
EXCHANGE = "NSE"
START_YEAR = 2015
COLOR_CUTOFF = 10  # max intensity cutoff (%)
# TradingView color theme
POS_COLOR = (8, 153, 129)    # #089981
NEG_COLOR = (242, 55, 69)    # #F23745
BG_COLOR = "#1e222d"
HEADER_BG = "rgba(128,128,128,0.2)"
TEXT_COLOR = "#d1d4dc"
LINE_COLOR = "rgba(128,128,128,0.3)"
MONTH_NAMES = ["Jan", "Feb", "Mar", "Apr", "May", "Jun",
               "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]

def calc_cell_color(value, cutoff=COLOR_CUTOFF):
    """Calculate cell background color matching TradingView's gradient logic."""
    if pd.isna(value):
        return "rgba(128,128,128,0.3)"
    base = POS_COLOR if value >= 0 else NEG_COLOR
    intensity = min(abs(value) / cutoff, 1.0)
    opacity = 0.10 + intensity * 0.40
    return f"rgba({base[0]},{base[1]},{base[2]},{opacity})"

def calc_pos_pct_color(value, cutoff=50):
    """Color for Pos% row: treat (value - 50) as the signed value."""
    if pd.isna(value):
        return "rgba(128,128,128,0.3)"
    shifted = value - 50
    base = POS_COLOR if shifted >= 0 else NEG_COLOR
    intensity = min(abs(shifted) / cutoff, 1.0)
    opacity = 0.10 + intensity * 0.40
    return f"rgba({base[0]},{base[1]},{base[2]},{opacity})"

def fetch_monthly_data(client, symbol, exchange, start_year):
    """Fetch daily data and resample to monthly close prices."""
    start_date = f"{start_year - 1}-12-01"
    end_date = pd.Timestamp.now().strftime("%Y-%m-%d")
    df = client.history(
        symbol=symbol, exchange=exchange,
        interval="D", start_date=start_date, end_date=end_date
    )
    if df is None or df.empty:
        raise ValueError("No data returned from API.")
    # Resample to monthly - use last close of each month
    monthly = df["close"].resample("ME").last().dropna()
    # Drop the current incomplete month
    today = pd.Timestamp.now(tz=monthly.index.tz)
    last_complete = (today.replace(day=1) - pd.Timedelta(days=1)).normalize()
    monthly = monthly[monthly.index <= last_complete]
    return monthly

def build_seasonality_matrix(monthly_close, start_year):
    """Build year x month matrix of monthly % returns."""
    returns = monthly_close.pct_change() * 100
    years = sorted(y for y in set(returns.index.year) if y >= start_year)
    matrix = pd.DataFrame(index=years, columns=range(1, 13), dtype=float)
    for dt, ret in returns.items():
        if dt.year >= start_year:
            matrix.loc[dt.year, dt.month] = ret
    return matrix

def build_heatmap_figure(matrix, symbol, exchange):
    """Build Plotly figure matching the TradingView seasonality heatmap."""
    years = list(matrix.index)
    n_years = len(years)
    avgs = [matrix[m].mean() for m in range(1, 13)]
    stdevs = [matrix[m].std(ddof=1) for m in range(1, 13)]
    pos_pcts = []
    for m in range(1, 13):
        col = matrix[m].dropna()
        pos_pcts.append(
            (col >= 0).sum() / len(col) * 100 if len(col) > 0 else float("nan")
        )
    header = ["Year"] + MONTH_NAMES
    n_rows = n_years + 4
    cell_values = [[] for _ in range(13)]
    cell_colors = [[] for _ in range(13)]
    # Year rows
    for year in years:
        cell_values[0].append(str(year))
        cell_colors[0].append(HEADER_BG)
        for m in range(1, 13):
            val = matrix.loc[year, m]
            if pd.isna(val):
                cell_values[m].append("NaN%")
                cell_colors[m].append("rgba(128,128,128,0.3)")
            else:
                cell_values[m].append(f"{val:.2f}%")
                cell_colors[m].append(calc_cell_color(val))
    # Divider
    for c in range(13):
        cell_values[c].append("")
        cell_colors[c].append(HEADER_BG)
    # Avgs
    cell_values[0].append("Avgs:")
    cell_colors[0].append(HEADER_BG)
    for m in range(1, 13):
        cell_values[m].append(f"{avgs[m-1]:.2f}%")
        cell_colors[m].append(calc_cell_color(avgs[m-1]))
    # StDev
    cell_values[0].append("StDev:")
    cell_colors[0].append(HEADER_BG)
    for m in range(1, 13):
        cell_values[m].append(f"{stdevs[m-1]:.2f}")
        cell_colors[m].append("rgba(128,128,128,0.2)")
    # Pos%
    cell_values[0].append("Pos%:")
    cell_colors[0].append(HEADER_BG)
    for m in range(1, 13):
        cell_values[m].append(f"{pos_pcts[m-1]:.0f}%")
        cell_colors[m].append(calc_pos_pct_color(pos_pcts[m-1]))
    fig = go.Figure(data=[go.Table(
        columnwidth=[80] + [100] * 12,
        header=dict(
            values=header,
            fill_color=HEADER_BG,
            font=dict(color=TEXT_COLOR, size=15,
                      family="Trebuchet MS, sans-serif"),
            align="center",
            line=dict(color=LINE_COLOR, width=1),
            height=40,
        ),
        cells=dict(
            values=cell_values,
            fill_color=cell_colors,
            font=dict(color=TEXT_COLOR, size=14,
                      family="Trebuchet MS, sans-serif"),
            align="center",
            line=dict(color=LINE_COLOR, width=1),
            height=36,
        ),
    )])
    fig.update_layout(
        title=dict(
            text=f"Seasonality - {symbol} ({exchange}) Monthly Returns",
            font=dict(color=TEXT_COLOR, size=16,
                      family="Trebuchet MS, sans-serif"),
            x=0.5,
        ),
        paper_bgcolor=BG_COLOR,
        margin=dict(l=10, r=10, t=50, b=10),
        height=max(400, 40 + n_rows * 36 + 60),
    )
    return fig

def main():
    client = api(api_key=API_KEY, host=HOST)
    print(f"Fetching daily data for {SYMBOL} on {EXCHANGE}...")
    monthly_close = fetch_monthly_data(client, SYMBOL, EXCHANGE, START_YEAR)
    print(f"Got {len(monthly_close)} monthly data points")
    matrix = build_seasonality_matrix(monthly_close, START_YEAR)
    print(f"Seasonality matrix: {matrix.shape[0]} years x {matrix.shape[1]} months")
    fig = build_heatmap_figure(matrix, SYMBOL, EXCHANGE)
    fig.show()
    print("Seasonality chart opened in browser.")

if __name__ == "__main__":
    main()

How It Works

Let’s walk through the key parts of the code:

Step 1 — Fetch and Resample Data

We use the OpenAlgo SDK’s client.history() method to pull daily closing prices. Since the API doesn't provide monthly candles directly, we resample the daily data to monthly frequency using pandas:

monthly = df["close"].resample("ME").last().dropna()

This gives us the last closing price of each completed month — exactly what TradingView uses for its monthly candles.

Step 2 — Calculate Monthly Returns

The monthly return is calculated as the percentage change from the previous month’s close to the current month’s close:

returns = monthly_close.pct_change() * 100

For example, if January’s close was 1000 and February’s close was 1050, the February return is +5.00%.

Step 3 — Build the Matrix

We organize the returns into a year-by-month matrix (rows = years, columns = Jan through Dec). This is the core data structure behind the heatmap.

Step 4 — Compute Summary Statistics

For each month column, we compute:

• Average: matrix[m].mean() — the mean return across all years
• Standard Deviation: matrix[m].std(ddof=1) — sample standard deviation (matching TradingView's calculation)
• Percent Positive: the fraction of non-NaN values that are >= 0

Step 5 — Color the Cells

The color logic mirrors TradingView’s approach. Each cell gets a green or red background with intensity proportional to the absolute return value, capped at a cutoff (default 10%):

base = POS_COLOR if value >= 0 else NEG_COLOR
intensity = min(abs(value) / cutoff, 1.0)
opacity = 0.10 + intensity * 0.40

Small moves appear as faint shading; large moves appear as deep, saturated color. This makes it easy to visually scan for extreme months.

Reading the Heatmap

Once you run the script, you’ll see a heatmap like this in your browser:

• Green cells indicate months where the stock gained value
• Red cells indicate months where the stock lost value
• Deeper colors mean larger moves; lighter colors mean smaller moves
• NaN% appears for months that haven’t completed yet (e.g., the current month)

What to Look For

Consistent columns of green: If October has been green for 9 out of 11 years (82% Pos%), that’s a strong seasonal bullish tendency. Consider timing long entries around late September.

Consistent columns of red: If September shows only 18% positive years, it’s historically the worst month. This might not be the best time to initiate new long positions.

High StDev months: Even if the average is positive, a high standard deviation means outcomes are unpredictable. Use these months for smaller position sizes.

Low StDev + High Pos%: This is the gold standard — a month that is consistently positive with low volatility. These months offer the most reliable seasonal edges.

Practical Applications

For Swing Traders

Use the Pos% row to identify the highest-probability months for your stock. Plan your entries and exits around these seasonal windows. If a stock has 100% positive Aprils over a decade, that’s a month where the odds are heavily in your favor.

For Portfolio Managers

Seasonality can guide asset allocation decisions. Rotate into sectors that historically outperform in the coming months. For instance, banking stocks in India often see strength in October (festive season lending) and weakness in September (quarter-end provisioning).

For Options Traders

StDev data directly informs volatility expectations. Months with historically high StDev suggest buying options (expecting large moves), while low StDev months favor selling options (expecting range-bound action).

For Algo Traders

Incorporate seasonal filters into your strategies. A simple rule like “only take long signals during months with Pos% > 60%” can meaningfully improve a strategy’s win rate without adding complexity.

Limitations to Keep in Mind

Seasonality is a statistical tendency, not a prediction. Here are the caveats:

1. Past patterns don’t guarantee future results. Black swan events, policy changes, and structural market shifts can override seasonal tendencies.
2. Sample size matters. A 10-year history gives you only 10 data points per month. That’s enough to spot strong tendencies but not enough for high statistical confidence. Be cautious with patterns based on fewer than 8–10 years of data.
3. Outliers can skew averages. A single month with a -35% return (like March 2020) can drag down the entire column average. Always look at Pos% alongside the average to catch this.
4. Seasonality works best as a filter, not a signal. Use it to time entries and exits within a broader strategy — not as the sole reason to trade.

Conclusion

Seasonality is a powerful analytical lens that most retail traders overlook. By studying how a stock behaves across different months, you gain an informational edge that complements technical and fundamental analysis.

With the OpenAlgo SDK and a few lines of Python, you can build a professional-grade seasonality heatmap that rivals TradingView’s built-in indicator — giving you the flexibility to analyze any stock, any exchange, and any time range you choose.

The code is available in the OpenAlgo examples directory. Modify the SYMBOL, EXCHANGE, and START_YEAR variables to analyze any instrument in your universe.

Built with OpenAlgo — the open-source algorithmic trading platform for Indian markets.

The Lark A1: The New King of Portability

• Ultra-Lightweight: Each transmitter weighs only 8g, making it virtually unnoticeable when clipped to a shirt.
• Feature-Rich App: Includes three Equalizer modes (Balanced, Low, Bright), three Reverb settings, and an auto-power-off function after 15 minutes to save battery.
• Price Point: Significantly more affordable than the M2, often starting at a much lower price point for creators on a budget.
• Mute Functionality: A double-tap on the transmitter allows you to mute individual microphones instantly.

• Basic Accessories: Unlike the M2, it does not come with extra mounting accessories.
• Windscreen Issues: The windscreen can easily fall off if it rubs against clothing as it lacks a locking mechanism.

The Lark M2: The Professional’s Reliable Choice

• Superior Range: Offers an impressive line-of-sight range of up to 1,000 feet (304.8m), compared to the A1’s 656.2 feet (200m).
• Audio Quality: Users note that the M2 provides a richer sound profile compared to the A1.
• Accessory Kit: Includes magnets and more robust mounting options out of the box.
• Fast Connection: The receiver pairs and starts up faster when plugged into devices like action cams.

• Lacks Auto-Off: Does not have the 15-minute auto-power-off feature found in the A1.
• Higher Price: It is roughly double the cost of the A1.

Side-by-Side Comparison

Who Should Choose Which Mic?

• For Mobile-Only Creators: The Lark A1 is the best fit. Its pill-shaped design, light weight, and mobile-specific app features (like Reverb for “bathroom singers”) make it perfect for TikTok, Reels, and casual vlogging.
• For Action & Long-Range Vlogging: The Lark M2 is preferred if you need to record subjects at a distance or need a faster, more reliable connection for action cameras like the DJI Action series.

Final Verdict

The post Hollyland Lark A1 vs. Lark M2: The Ultimate Compact Mic Comparison appeared first on IB Computing.

The New Rules: Names and Numbers

Several key pieces of legislation are leading this global trend:

• Brazil: The Digital Statute for Children and Adolescents (Law No. 15,211/2025) is one of the most advanced child-protection frameworks globally.
• California: The Digital Age Assurance Act (Assembly Bill No. 1043) requires OS providers to provide an age signal to applications.
• New York: The proposed Senate Bill S8102A is even more stringent, explicitly forbidding simple self-reporting of age.
• Colorado: Senate Bill 26-051 closely mirrors California’s approach, requiring OS-level age bracket reporting.

When Must Systems Comply?

The clock is already ticking for developers and manufacturers:

• Brazil: The decrees regulating the law entered into force on March 17, 2026.
• California: The act becomes fully operative on January 1, 2027.
• Colorado: This legislation is scheduled to take effect on January 1, 2028.

The High Price of Non-Compliance

The penalties for failing to implement these systems are substantial. In Brazil, companies can face fines of approximately $9.5 million (R$50 million) per violation, where each individual user who is not presented with the required verification counts as a separate failure. California imposes civil penalties of $2,500 for negligent violations and $7,500 for intentional ones per affected child. New York’s proposed bill suggests even higher penalties of $10,000 per violation.

How Verification Works Technically

Under these laws, the OS provider—the entity that develops or controls the software—is responsible for the verification process.

1. Interface: The OS must provide an interface during account setup for the “account holder” (an adult or guardian) to indicate the user’s age or birth date.
2. Digital Signal: The OS then generates a digital signal via a secure API.
3. Age Brackets: This signal informs apps which of four brackets the user falls into: under 13, 13–16, 16–18, or 18+.

The Privacy Debate: Protection or Surveillance?

The most significant criticism of these laws centers on privacy. Critics, including over 400 computer scientists, argue that these mandates create a massive surveillance infrastructure. Collecting birth dates—which are Personally Identifiable Information (PII)—creates a centralized database of children, making them prime targets for data breaches. Furthermore, the U.S. Federal Trade Commission (FTC) has suggested it might ignore certain privacy violations (COPPA) to “incentivize” the use of these age-verification tools, a move critics say effectively encourages the creation of child databases.

The View from System76

Carl Richell, CEO of Linux hardware manufacturer System76, has been a vocal critic. He argues that these laws are ineffective because children are tech-savvy and will easily find workarounds, such as using virtual machines or VPNs to bypass restrictions. Richell also worries that these laws limit a child’s ability to explore technology, noting that many of the world’s best programmers started by experimenting with operating systems as children. He advocates for education about “digital abundance” rather than “nerfing” the internet through failed legislative restrictions.

Linux and the FOSS Response

The decentralized nature of Linux makes compliance particularly difficult.

• systemd: The project recently merged a birthDate field into its JSON user records to serve as a data source for age signals, though this move was met with significant community pushback.
• Ageless Linux: This project offers a script to explicitly refuse age collection. They argue that if no age data is collected, no one can be identified as a “child,” meaning there is no “affected user” to trigger a fine—essentially a “logic puzzle” loophole.
• GrapheneOS: The privacy-focused Android fork has outright refused to comply, stating it will never require personal information or accounts. They have expressed a willingness to be banned from certain regions rather than compromise their principles.
• MidnightBSD: This project updated its distribution terms to ban users in regions with these laws, stating their software will not include age checks.

Mainstream Readiness

In contrast to the struggle within the Linux community, mainstream providers like Apple, Google, and Microsoft are expected to comply with almost no additional cost. These companies already maintain centralized account systems with existing parental controls and age-gating infrastructure.

Conclusion

While these laws are born from a genuine desire to protect children, they represent a fundamental shift in how we interact with our devices. For many, the choice may soon be between a “compliant” system that tracks personal data and a “nerfed” internet experience that prioritizes anonymity. As these deadlines approach, the tension between digital safety and individual liberty has never been higher.

The post Age Verification Laws: Is This the End of Privacy for Linux and Beyond? appeared first on IB Computing.

Anyways, I recently got my hands on a used thinkpad X230 for around 60$.

The machine is great, the keyboard even better, but the TN panel is genuinely unusable.

For a while, I used it like a laptop as intended, but I can only do so much with a TN Panel whose max brightness is my Pixel’s 10% brightness.

Instead, I decided to cut my losses and just remove the display altogether, and use the thinkpad as a keyboard with my monitor.

The Hardware

Doing this was pretty simple, I just had to open up the back of the laptop, remove the hinge screws, and then slowly disconnect all the wires before removing the display assembly.

Additionally, I changed the Wi-Fi card to an atheros one for good measure. (God bless Libreboot)

Main issue with this, was that the Thinklight is literally impossible to remove from the casing, so I kinda had to snap that wire out.

I also removed the two wifi antennae that are glued to the display casing, and kept it outside so wifi continues to work properly.

I kept the webcam too, removed from the case and double tape’d to my monitor - since somehow this ancient laptop seems to have a better camera than my modern laptop.

After doing all this, and very-safely electrical taping all the extra wires if i ever decide to put a new display on this, I got a think-slab :D

However, I had to add a few cmdline arguments on grub to make it work.

/etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iomem=relaxed i915.modeset=1 video=LVDS-1:d video=VDA-1:e"

The first 3 are common grub parameters that you always have, while the last 2 are the special ones you need to add the :d parameter disables LVDS-1 (internal display), and the :e parameter enables VDA-1 (in my case, the external display)

An update-grub later, everything magically started appearing on my monitor!

The software

This worked great for research and other random stuff I did, but a laptop from 2012 can only do so much in terms of computing.

To remedy this, I decided to just use the thinkpad keyboard with my modern laptop using software KVM. Since my monitor has both VGA and HDMI, I was able to connect my modern laptop to the HDMI port, and the thinkpad to VGA.

Though I was just planning on using barrier like I did year’s ago, I decided to go with Lan-Mouse this time.

It is a rust-based application similar to synergy and barrier, but with a proper gtk UI and supposedly better performance.

Now, since I have a wireless card that’s older than me on the thinkpad, I had to do some ethernet magic for lan mouse if I wanted any sort of real performance. Software KVM is a high-bandwidth task after all.

So I connected an ethernet cable between the two laptops, and set it up as follows

# On Modern laptop
nmcli con add type ethernet ifname enp2s0 ip4 192.168.50.1/24
# On Thinkpad
nmcli con add type ethernet ifname eno0 ip4 192.168.50.2/24

And magically, I have a Gigabit connection between the two laptops for Lan-Mouse to work through. Lan mouse is pretty intuitive to setup, so im not covering it here.

In conclusion, this one day’s work turned out pretty well for me. I now have a speedy “slabtop” for any research or minor work that I have, and it doesn’t even take 5 minutes for this setup to convert into a high-performance workstation either.

I’m still using the X230 without lan-mouse for home usage, when I’m too lazy to get my laptop out of my bag, but this setup really helps me when I need to get some real programming work done, which is 10x harder without such a comfy keyboard like that of the X230.

Continuing from the last post, Badri and I took a flight from the Brunei International Airport to Kuala Lumpur on the 12th of December 2024. We reached Kuala Lumpur in the evening.

After arriving at the airport, we went through immigration. In a previous post, I mentioned that we had put our stuff in lockers at the TBS bus terminal in Kuala Lumpur. Therefore, we had to go there.

The locker was automated and required us to enter the PIN we had set. Upon entering the PIN, the locker wasn’t getting unlocked. After trying this for 10-15 minutes without any luck, we tried getting some help as there the lockers weren’t under supervision.

So, I roamed around and found a staff member, reporting that our lockers weren’t getting unlocked. They called the person who was in-charge of the lockers. He came to us in a few minutes and used their admin access to open the locker. We were supposed to pay for using the lockers by putting the banknotes inside through a slot. However, as the machine wasn’t working, we gave the amount for the use of our locker service to that person instead.

We soon went back to the KL airport to catch our morning flight to Ho Chi Minh City in Vietnam. At the flight counter, we were afraid we would have to pay extra as our luggage surpassed the allowed weight limit. This one was also a budget airline—AirAsia—and our tickets didn’t include a check-in bag.

Generally, passengers from countries requiring a visa to visit Vietnam (such as India) require going to the airline and showing their visa to get the boarding pass. However, when we went to the AirAsia counter at the Kuala Lumpur airport, they didn’t weigh our bags and asked us to get our boarding passes from an automated kiosk. So, we got our boarding passes printed and proceeded to the airport security.

While clearing the airport security, a lotion I bought from Singapore was confiscated because it was 200 mL, exceeding the limit of 100 mL per bottle. Had that 200 mL liquid been in two different bottles of 100 mL each, I would have been allowed to take it in my carry-on bag, but a single 200 mL bottle wasn’t! I was allowed to keep it in the check-in bag, but I didn’t have it included in my ticket. Huh, airports and their weird rules :( The lotion was an expensive one, so having it thrown away did ruin my mood.

Overview

We started our Vietnam trip from Ho Chi Minh City in the south on the 13th of December 2024 and finished it in Hanoi in the north on the 20th of December. We traveled from Ho Chi Minh City to Hanoi mostly by train, except for a hundred or so kilometers by bus, in chunks. On the way, we visited Nha Trang, Hoi An, and Hue. The distance between Ho Chi Minh City and Hanoi is 1700 km.

For your reference, here are those places labeled on Vietnam’s map.

Ho Chi Minh City

We landed in Ho Chi Minh City early morning on the 13th of December 2024. I was tired and sleepy as I hadn’t gotten a good night’s sleep. After going through immigration, we went to a currency exchange counter to get Vietnamese Dong. Unlike other countries on this trip, money exchange counters in Vietnam didn’t accept Indian rupees. Therefore, we exchanged euros to get Vietnamese dong at the airport.

After getting out of the airport, we took a bus to the city center. It was 15,000 dongs—approximately 50 Indian rupees. Our plan was to meet Badri’s friend and stay the night at his apartment.

So we went to a café nearby and bought a coffee for each of us for 75,000 dongs. We went upstairs and sat for a while. The Wi-Fi password was mentioned on our bill. During the trip, I found out about the café culture of Vietnam. They have their own coffee brands (such as Highlands Coffee), and you can sit down at any of the cafés for work or wait for the rain to stop. It rained a lot while we were there, so we did use these cafés for that purpose.

Badri’s friend met us there, and we roamed around the area a bit, which included roaming inside a beautiful park. Then Badri’s friend took us to a restaurant. Because I do not eat meat, he took us to a vegan restaurant. Having been to four Southeast Asian countries at this point (excluding Vietnam), I was under the impression that there wouldn’t be a lot of things for my diet in Vietnam.

However, I was pleasantly surprised at the restaurant. I found all the dishes to be tasty, especially their signature noodles called Pho. I liked another dish so much that I tracked down the restaurant again with Badri using the geotagged image of the bill I had taken earler to have it again. As a tip for vegans coming to Vietnam, the places having the letters “Chay” (without any accented letters) in their name are vegan only.

In the night, we went to a supermarket where I got myself some oranges and guavas. Then, we went to a Japanese restaurant where I didn’t have anything, as there was no vegetarian option available for me. Then we took a free bus to the place to Badri’s friend’s apartment. The construction company that built the apartment also runs this free bus service from their residential area to different parts of the city as a way of promoting their apartments. Anyone can take the bus, not just residents.

The next day, we took the free bus back to the city center and checked in to a hostel for a night. We took two beds in dormitories, which were 88,000 dongs (270 rupees) for each bed for a night. In Vietnam, if you can spend around 300 rupees per night, you can get a bed in a decent hostel.

Train from Ho Chi Minh City to Nha Trang

On the night of the 15th of December 2024, we boarded a train from Ho Chi Minh City to Nha Trang. The ticket for each of us was 519,000 dongs (1600 Indian rupees). The train name was SNT2. When we reached the Ho Chi Minh City train station, we noticed that the station was rather small by Indian standards.

After entering the train station, we went inside to the first platform, where the tickets were checked by a staff member. Ho Chi Minh City was the originating station for our train, so our train was already standing at the station. We had to cross the railway tracks on foot to reach the platform our train was on. Then we located our coach, where a ticket inspector was standing at the gate. He let us in after checking our tickets. In all these instances, we just had to show our digital boarding pass which we had received by email.

Unlike Indian trains, the train didn’t have side berths. Additionally, I liked the fact that it had a dedicated space to put our bags in, which was very convenient. The train departed from Ho Chi Minh City at 21:05 and arrived in Nha Trang at 05:30 in the morning.

Nha Trang

Nha Trang is a coastal place, and we planned to go to a beach. We figured out that the bus to the airport goes can drop us near the beach. Therefore, we went to the bus station to get to the airport bus. The bus station was walking distance from the railway station. So, we decided to walk.

On the way, we stopped at a small shop for a coffee. The shop also gave a complimentary cup of green tea along with the coffee. I found out later that it is common for local shops to give a cup of complimentary green tea in Vietnam.

Soon we reached the bus station and took a bus to the beach. It was 65,000 dongs (₹200). After getting down from the bus, I had coconut water and some eggs at a small local place.

Then we went to the beach, but nobody else was there. We spent some time there and went back to the place where the bus dropped us as it started raining. We couldn’t find a bus for some time. A taxi driver approached us and agreed to take us to the city center for 200,000 dongs (₹650). For reference, the place where he dropped us was 35 km from the place we took the taxi. Taxi fares in Vietnam were also cheap!

Nha Trang was a beautiful place, and so we roamed around for a while. Then we stopped at a Highlands Coffee branch for a while. Since Christmas was coming up, the café had a Christmas tree, and I liked the Christmas vibes. They were playing Mariah Carey’s All I Want for Christmas Is You.

During the evening, we went to a local place to eat. The place mentioned “Chay” in its name, and you know what it means—it was a vegan place. There was a man there and no other customers. I don’t remember the names of the dishes we ordered, but it was a bowl of soupy noodles and a bowl of dry noodles. They were very tasty. To top that off, the meal was a total of 55,000 dongs (₹180) for both of us.

The host was welcoming and friendly. We had a nice conversation with the host. In Vietnam, restaurants give chopsticks to eat noodles. While Badri was good at using them, I wasn’t. So, the host of this restaurant helped me in using chopsticks. Although my technique was not perfect and I take a bit of time, I could now eat solely with chopsticks.

Our plan was to take a night bus to Hoi An, and we were hoping to find a bus stand. However, we couldn’t find one. Asking around about the pickup location of the Hoi An bus led us to many different locations. Finally, we ended up at a bus booking agency’s office where we found out that there were no tickets available for Hoi An.

At this point, we gave up on booking the bus and searched for trains instead. As we didn’t have a local SIM, we asked the agency to let us connect to their Wi-Fi so that we could look for trains. They were kind enough to let us do that. It also seemed like they were going to close the office in like 10 minutes.

Unfortunately, all the sleeper berths were booked from Nha Trang till Hoi An on the next train with only seating berths being available. It takes around 10 hours, so I wasn’t comfortable traveling on seating berths.

Here I came up with the idea to look for sleeper berths from an intermediate stop. Fortunately, there were sleeper berths available from the next stop, Ninh Hòa. Therefore, we booked a seating berth from Nha Trang to Ninh Hòa and a sleeper berth from Ninh Hòa to Trà Kiệu (the nearest railway station from Hoi An). The train name was SE6, and it was a total of 500,000 dongs per person (₹1600 per person).

So, we went to the Nha Trang railway station and boarded the train. We had to spend 40 minutes seated for the train to reach the next stop before we could go to our sleeper berths. Badri had some friendly co-passengers on that trip who gave him Saigon beer and some crispy papad-like thing. They offered me as well, but I thought it was non-veg, so I declined it.

Hoi An

On the morning of 17th December 2024, we got down at the Trà Kiệu station at around 09:30. Our hostel was in Hoi An, which was around 22 km from the station. There was no public transport to get there.

Instead, there was a taxi driver at the train platform. We told him the name of our hostel, and he quoted 270,000 dongs (around ₹850). We said it was too expensive for us, so he agreed to bargain at 250,000 dongs. At this point, we told him that we could give him no more than 200,000 dongs, but he didn’t agree.

Badri tried a trick. He asked the driver to show us prices in the Grab app (a popular taxi booking app in Southeast Asia). Unfortunately, the Grab app showed 258,000 dongs, which was more than the fare the driver agreed to.

So we walked away as if we had so many options (we didn’t!) to reach the hostel. We got out of the station and stopped at a small shop outside to have some coffee. As is customary in Vietnam, we got a complimentary green tea here as well.

That taxi driver also joined us and sat in that shop. He started talking with the locals in the shop in the local language. The taxi driver was insistent on taking us to Hoi An for 250,000 dongs. At this point, Badri told the taxi driver (by the use of translation software) that we usually use public transport during our trips, and we aren’t used to paying high prices to get around. So, he can drop us somewhere in Hoi An for 200,000 dongs as we don’t mind walking a bit to reach our hotel.

After reading this, the taxi driver agreed to take us to our hostel for 200,000 dongs (₹660). He also had me take a picture with Badri after this. I think such a bargain tactic would not work in India.

The nice thing we noticed in Vietnam is, once bargaining is done and the deal is settled, people don’t try to bargain more or keep on talking about the subject. Before the deal, the driver was being somewhat insistent and argumentative, but after the deal was done, it was as if no argument had happened at all.

We were treated to some beautiful scenery on the way to our hostel. Soon we reached our place and completed all the formalities for checking-in. During the time our room was being prepared for check-in, we had an egg sandwich with coffee in the hotel. I found the egg sandwich very tasty. The bread looked like the French baguette. The hostel was ₹240 per night for each of us.

The name of the hostel was Bana Spa. We liked staying here and we can recommend it if you find yourself there. It is operated by a family.

We also rented a bicycle for each of us—25,000 dongs per day (₹80)—and explored the old town during the evening. Hoi An is popular for Vietnamese silk. Tourists come here to buy fabric and get it done by the tailor. The buildings here looked old, and they were painted in yellow with a gabled roof.

Here, I also had egg coffee for the first time, and I liked it. Egg coffee is a delicacy of Hanoi, but you can get it in other parts of Vietnam. If you find yourself in Vietnam, then I recommend you try egg coffee. We also bought some cool T-shirts and other souvenirs, such as a Vietnamese hat, from here.

Hue

The next day—the 18th of December 2024—we went to Hue by bus. As we could not take a bus on our own in Nha Trang, we asked the hostel to book it for us this time. We booked it a day before, and they told us to be ready by 07:00 in the morning. At 07:00, a minibus arrived, which took us to a bus agency’s office. There we waited for a few minutes and got into the bus to Hue.

The bus had sleeper seats, so I took the opportunity to catch some sleep. The ride was comfortable, so I am assuming the roads were good. In a couple of hours, we reached Hue. Again, we went to Highlands Coffee to have some coffee, charge our phones, and use the internet, not to mention using the bathrooms.

During the afternoon, we went to a local restaurant named Quán Chay Thanh Liễu. It was a vegan restaurant (remember the thing I mentioned earlier about “Chay” being in the name?). On the way, we had a steamed dumpling shaped like a momo called banh bao from a street vendor. It wasn’t very good, but I found it worthwhile.

At the restaurant, we ordered a hot pot. First, they brought noodles and a gas stove. Then came the stock and our gas stove was turned on. The stock was kept simmering on the stove. Then, we had it bit by bit with the noodles. A big hot pot at this place costs 50,000 dongs (₹170). Then we had bánh cuốn. These were steamed rolls made of rice flour for 10,000 dongs (₹33).

Restaurants in Vietnam usually add photos of the meals in their menu or write a description in English. So, even though the dish names were Vietnamese, we had no problems in ordering food there. In addition, all the places we went to provided free Wi-Fi. They either mention the Wi-Fi password on the bill, on the menu or paste it on the wall. This made our trip smoother without getting a local SIM.

We had booked the train SE20 for Hanoi, which had a departure time of 20:41 from Hue. This one was 948,000 dongs (₹3100) for myself and 870,000 dongs (₹2900) for Badri. My ticket was pricier than Badri’s because I got a lower berth. Our train was late by half an hour, so we waited in the common area of the station. After the train arrived, we got inside and took our seats.

The cabin had four berths—two upper and two lower, similar to India’s First AC class. The ticket inspector came to us and offered us the whole cabin (two additional berths) for 300,000 dongs (₹1,000), which we declined. However, this hinted at the other two seats not being reserved. Eventually, we had the whole cabin to ourselves, as nobody else showed up for the other two berths. It was a 14-hour journey, and I got a good sleep.

Hanoi

On the morning of the 19th of December 2024, we reached Vietnam’s capital, Hanoi. We had booked a private hotel room for ₹800. It was 1 km from the Hanoi Airport. However, it was pretty far from the railway station. So, we roamed around in the city and went to the hotel in the evening.

First, we walked to a place and had egg coffee with egg sandwiches. Then we went to Hanoi Train Street, which was walking distance from the train station. After clicking some pictures at the train street, we went to a museum nearby. Upon reaching there, we found out that it was closed.

Then we went shopping for jackets, as Hanoi was cold compared to other parts of Vietnam we had been to, and since many of them are manufactured in Vietnam, we thought they would be cheaper. I liked some jackets, but they were not my size. Eventually, we didn’t buy anything at the clothes shop.

In the evening, I bought a Vietnamese-styled phin coffee filter and coffee powder from Highlands Coffee. We spent a lot of time in their cafes, so it made sense to buy some souvenirs from there. Badri bought a few coffee filters for his family at Trung Nguyen, where I also bought another filter.

We had dinner at a local place where we had pho and banh it. Bahn it was served packed in banana leaves and it was made of sticky rice.

Next, we went to Hanoi railway station to catch a bus to the airport since our hotel was 1 km from the airport. The locals there helped us take the bus. It took like an hour to get to the airport. We saw on OpenStreetMap that we can take a bus from there to the hotel, but we could not find it. So we walked to our hotel instead.

It was a decent hotel room for ₹800 for a night. We went outside to explore the area and had egg sandwiches and egg coffee at a local place. Again, we were given a complimentary green tea. We went to this place like three times. We had practically become regulars by the time we left.

The next day— 20th of December 2024 — we took a bus to the airport and boarded our flight to Delhi.

Credits: Thanks Badri, Kishy and Richard for proofreading.

வேண்டும் வேண்டும் என வேண்டும் வெற்றிகள்
என்னிடம் போதும் போதும் என கூறும் தோல்விகள்

சிறுசிறு வெற்றிக்கும் மகிழ்ச்சி அடைந்தேன் அன்று
தினமும் மூன்று...கிடைக்குமா? சற்று பார்ப்போம் முயன்று 

தீராக் காதல் அதீத அன்பினால் தினமும் மோதல்
போதும் எல்லாம் என்று முடிக்கையில் மீண்டும் பூக்கும் சிறு காதல்

ஊடலும் முடியவில்லை கோபமும் குறையவில்லை
முயற்சி செய்கிறேன் அமைதியாக! சந்தேகங்கள் எழாமலில்லை

மதிமுகம் காணவந்தவன் நிம்மதி தேடியே அலைகிறேன்
விதை விதைக்கையில் மரம் தழைக்குமா யோசிக்கிறேன்

பிடித்ததை செய்கையில் பித்துப் பிடித்து போகிறது
கதையை முடிக்க செல்கையில் கவிதையோ தொடங்குகிறது

                   -- -- 
எழுத்து : ஹரிஹரன் உமாபதி
வரைகலை: செய்யறிவு மற்றும் கிம்ப்(Gimp)
 

These are not random bugs. They are symptoms of an architectural pattern that every trading platform eventually outgrows.

This is the story of how OpenAlgo adopted event driven architecture, what it is, why algorithmic trading platforms need it, and what it unlocks for strategy level position tracking, risk management, and beyond.

What Is Event Driven Architecture?

Let us start with an analogy every trader understands.

The Trading Floor Analogy

Imagine a trading floor in the 1980s. A floor trader executes a buy order, then personally walks to the risk desk to report it, then walks to the settlement desk, then calls the back office, then updates the position board.

If the risk desk is on a coffee break, the trader stands there waiting. The settlement desk does not get the information until the risk desk conversation is done. If the trader forgets to update the position board, which happens when things get hectic, nobody notices until reconciliation.

Now imagine a modern electronic exchange. The trader submits the order. The exchange publishes a fill message to the wire. The risk system reads it. The settlement system reads it. The position board reads it. The compliance system reads it. Each one independently, simultaneously, without knowing about the others.

The trader does not walk to five desks. The trader announces “this happened” once. Everyone who cares is listening.

That is event driven architecture.

The Two Models

Direct calls, request response:

Order Service -> calls Logger
             -> calls Dashboard
             -> calls Telegram
             -> calls Risk Monitor

The order service knows about every consumer. Adding a new one means editing the order service. If one consumer is slow, it slows down everyone after it. If one crashes, the chain breaks.

Event driven, publish subscribe:

Order Service -> publishes "order.placed" event
                    |
         |----------|----------|----------|
         v          v          v          v
      Logger    Dashboard   Telegram   Risk Monitor

The order service does not know who is listening. Each consumer subscribes independently. Adding a new one requires zero changes to the order service. If Telegram is down, the logger and dashboard still work. If you add a risk monitor next month, you do not touch a single line of order code.

The technical term for this is decoupling, and for trading systems, it is not a nice to have. It is essential.

Why Algo Trading Platforms Need Event Driven Architecture

Trading platforms are not typical web apps. The order pipeline has unique properties that make direct function calls progressively more painful as the system grows.

1. The Order Pipeline Touches Everything

When an order is placed, the entire system needs to know:

• The database needs to log it, for audit trails and compliance
• The dashboard needs to refresh, so you see the result instantly
• Your phone needs to buzz, Telegram or WhatsApp alert
• The position tracker needs to update, so you know what you hold
• The risk engine needs to check, are you exceeding exposure limits
• The P & L calculator needs to recalculate, real time profit and loss
• The analytics engine needs to record, win rate, average trade, drawdown

That is seven consumers for a single order event. Wire them directly, and your order function becomes a 200 line monster that imports half the codebase. Event driven architecture lets the order function stay clean: place the order, announce what happened, return the result. Seven consumers handle the rest.

2. Your Broker Drops Your Strategy Identity

This is a problem unique to Indian markets and algo platforms like OpenAlgo.

When you send an order to Zerodha, Angel One, Fyers, Dhan, or any Indian broker, you include a strategy field, "Iron Condor", "Momentum Scanner", "Mean Revert". The broker ignores it. It does not store it. It does not return it.

When you later ask the broker for your positions:

NIFTY: +65 lots
SBIN: +100 shares

Which strategy holds those? If you are running “Momentum” with +100 SBIN and “Mean Revert” with 50 SBIN short, the broker shows you +50 SBIN. The per strategy breakdown is gone.

The only moment you can capture which strategy owns which order is at placement time. After that, the strategy tag disappears at the broker boundary forever.

Event driven architecture captures that moment. Every order event carries the strategy name. Any system that subscribes, position tracker, risk manager, analytics, gets the strategy identity preserved.

3. Live Trading and Sandbox Trading Must Behave Identically

Every serious algo trader tests strategies in a sandbox before going live. OpenAlgo’s Analyzer mode provides sandbox trading with sandbox capital.

But here is the challenge: when the same order function handles both live and sandbox mode, the side effects need to differ:

Live mode

• Log destination: order_logs table
• Dashboard event: order_event
• Telegram prefix: LIVE MODE, Real Order

Sandbox mode

• Log destination: analyzer_logs table
• Dashboard event: analyzer_update
• Telegram prefix: ANALYZE MODE, No Real Order

With direct calls, you need if branches for every side effect in every order service. With event driven architecture, the event carries a mode field, and each subscriber knows what to do:

if event.mode == "analyze":
    # sandbox path
else:
    # live path

One check per subscriber, not per service. The order service simply sets the mode and publishes.

4. Batch Orders Need Different Notification Semantics

A single order needs one notification. A basket of 20 orders needs one summary notification, not 20 individual ones. A split order breaking 1000 shares into 50 share chunks needs a summary, not 20 alerts.

With direct calls, every batch service has to manually suppress per order notifications and emit its own summary, a pattern that is easy to get wrong. With events, the batch service publishes one BasketCompletedEvent at the end. Individual sub orders publish nothing. The notification logic lives in one place.

5. Failure Isolation Is Critical When Money Is Involved

If Telegram’s API is slow and your notification call is inline with order placement, one of two things happens:

• The order response is delayed while waiting for Telegram, bad for latency
• The Telegram call fails, and depending on your error handling, the error might propagate up and make it look like the order failed, catastrophic for trust

Event driven architecture isolates failures by design. Each subscriber runs independently. Telegram is down? The Telegram subscriber logs an error. Your order still gets logged. Your dashboard still updates. Your position tracker still records the trade. No single subscriber failure can affect the order pipeline or any other subscriber.

How OpenAlgo Adopted Event Driven Architecture

What We Started With

OpenAlgo supports ten order types: place order, smart order, basket order, split order, options order, multi leg options, modify, cancel, cancel all, and close position.

Each service had three hardcoded side effects after every broker call:

# After the broker confirmed the order:
executor.submit(async_log_order, "placeorder", request_data, response)      # Log it
socketio.start_background_task(socketio.emit, "order_event", {...})         # Update dashboard
socketio.start_background_task(telegram_alert_service.send_order_alert, ...) # Alert phone

These three lines, with variations, appeared in every order service, in every success path, every failure path, and every sandbox path. That is 50 plus dispatch points across 18 files.

The Bugs We Found

• Close Position in Analyzer mode: Dead if False: code block. Positions closed silently, no log, no Telegram, no dashboard update
• Modify and Cancel in Analyzer mode: Telegram alerts were skipped entirely, while live mode sent them
• Basket order with 20 legs: 21 Telegram alerts and 21 log entries instead of 1
• Options multi order in Analyzer: Emitted order_event, live mode event, instead of analyzer_update, confusing the Analyzer UI
• UI Close Position button: Called the broker directly, bypassing the service layer. No order was ever logged
• API key in error logs: On validation failures, the raw API key was written to the log database

Every one of these bugs existed because side effects were scattered across files instead of centralized. No single file was “wrong”, the bugs emerged from inconsistencies between files.

The Event Bus We Built

We built a lightweight, in process event bus in about 60 lines of Python. No Redis. No Kafka. No external infrastructure.

How it works:

1. Order services publish typed events: OrderPlacedEvent, BasketCompletedEvent, OrderCancelledEvent, and others. Each event carries the mode, live or analyze, the strategy name, the request and response data, and the API key for notifications.
2. The event bus routes by topic: Topics like "order.placed", "basket.completed", "order.cancelled" determine which subscribers receive the event.
3. Subscribers handle one concern each:

• Log subscriber: Writes to order_logs, live, or analyzer_logs, analyze
• SocketIO subscriber: Emits the correct dashboard event, 8 different event names depending on operation and mode
• Telegram subscriber: Sends formatted alerts with detailed order information

1. Everything is async and isolated: A shared thread pool dispatches callbacks. One subscriber crashing does not affect the others.

Before and After

Before, the order service imported and called three systems directly:

# place_order_service.py (old)
from database.apilog_db import async_log_order, executor
from extensions import socketio
from services.telegram_alert_service import telegram_alert_service

def place_order_with_auth(order_data, auth_token, broker, original_data):
    # ... broker call ...
    if res.status == 200:
        executor.submit(async_log_order, "placeorder", request_data, response)
        socketio.start_background_task(socketio.emit, "order_event", {...})
        socketio.start_background_task(telegram_alert_service.send_order_alert, ...)

After, one line, one event:

# place_order_service.py (new)
from events import OrderPlacedEvent
from utils.event_bus import bus

def place_order_with_auth(order_data, auth_token, broker, original_data):
    # ... broker call ...
    if res.status == 200:
        bus.publish(OrderPlacedEvent(
            mode="live",
            api_type="placeorder",
            strategy=order_data.get("strategy", ""),
            symbol=order_data["symbol"],
            orderid=str(order_id),
            request_data=cleaned_request,
            response_data=response,
            api_key=api_key,
        ))

The service does not know who listens. It does not import logging, SocketIO, or Telegram. It announces what happened and moves on.

The Impact

• Side effect dispatch points: Before, 50 plus across 18 files. After, 15 bus.publish() calls across 10 services
• Files that know about logging: Before, 18. After, 1, log_subscriber.py
• Files that know about Telegram: Before, 12. After, 1, telegram_subscriber.py
• Files that know about SocketIO events: Before, 14. After, 1, socketio_subscriber.py
• Thread pools for side effects: Before, 3 pools, 25 threads. After, 1 pool, 10 threads
• Bugs from inconsistent side effects: Before, 6 known. After, 0

How Modularity Is Maintained

The event bus enforces a clean separation of concerns through a simple rule: publishers do not know about subscribers, and subscribers do not know about each other.

The File Structure

Order Services (publishers)           Event Bus              Subscribers (consumers)
─────────────────────────             ─────────              ────────────────────────
services/place_order_service.py    |                    |  subscribers/log_subscriber.py
services/basket_order_service.py   |                    |  subscribers/socketio_subscriber.py
services/split_order_service.py    | -> bus.publish() -> |  subscribers/telegram_subscriber.py
services/options_multiorder_...    |                    |  subscribers/strategy_store.py (future)
services/cancel_order_service.py   |                    |  subscribers/risk_manager.py (future)
services/close_position_service.py |                    |

Each column is independent. You can modify a subscriber without touching any service. You can add a service without touching any subscriber. The event bus in the middle is the only shared contract, and it is a 60 line class that never needs to change.

The Contract: Typed Events

Publishers and subscribers agree on the event schema, nothing else:

@dataclass
class OrderPlacedEvent(OrderEvent):
    topic: str = "order.placed"
    mode: str = "live"          # "live" or "analyze"
    api_type: str = ""          # "placeorder", "basketorder", etc.
    strategy: str = ""
    symbol: str = ""
    exchange: str = ""
    action: str = ""            # "BUY" or "SELL"
    quantity: int = 0
    orderid: str = ""
    request_data: dict = {}     # for logging, apikey stripped
    response_data: dict = {}    # for logging
    api_key: str = ""           # for Telegram username resolution

This is the contract. The publisher fills it. The subscriber reads it. They never import each other.

Adding a New Consumer: One File, One Line

Want to add a Discord notification alongside Telegram? Write a file, register it:

# subscribers/discord_subscriber.py
def on_order_placed(event):
    send_discord_webhook(f"Order placed: {event.symbol} {event.action}")

# subscribers/__init__.py, add one line:
bus.subscribe("order.placed", discord_subscriber.on_order_placed)

No order service changes. No existing subscriber changes. No deployment coordination. This is the power of decoupling.

The Future: Strategy Level Intelligence

The event bus is not just a code cleanup. It is the foundation for features that were architecturally impossible with hardcoded side effects.

Strategy Level Positions

Today, OpenAlgo shows positions from the broker, account level, no strategy breakdown.

Tomorrow, a new subscriber will listen to order events and maintain per strategy positions:

Strategy: Iron Condor
  NIFTY30MAR2623650CE  BUY   +65  @ 120.50
  NIFTY30MAR2622650PE  BUY   +65  @ 95.30
  NIFTY30MAR2623400CE  SELL  -65  @ 155.20
  NIFTY30MAR2622900PE  SELL  -65  @ 110.80
  Net P&L: +2,450

Strategy: Momentum Scanner
  SBIN      BUY  +100  @ 420.50   P&L: +1,250
  RELIANCE  BUY  +50   @ 1305.00  P&L: -430

This is just a new subscriber reading order.placed events. The order services do not change.

Strategy Level Orderbook and Tradebook

The broker’s orderbook has no strategy column. But every order event carries the strategy field. A subscriber can write these to an internal table, giving you:

• Filter orders by strategy
• See trade history per strategy
• Know which strategy generated which fills

Strategy Level Risk Management

With per strategy positions built from events, risk management becomes possible per strategy:

• Stoploss per strategy: “Exit all Iron Condor legs if net P and L drops below 5,000 loss”
• Target per strategy: “Close Momentum positions at +2 percent return”
• Trailing stoploss per strategy: “Trail SBIN stop by 10 points as price rises”
• Max position size per strategy: “Momentum Scanner cannot hold more than 500 shares of any symbol”
• Daily loss limit per strategy: “If Mean Revert loses 10,000 in a day, stop trading”

The risk manager subscribes to order events, to know what each strategy holds, and market data ticks, to monitor prices. When a limit is breached, it publishes a RiskExitEvent, which triggers an exit order through the same event bus.

The Extensibility Pattern

Every future feature follows the same three steps:

1. Create a new file in subscribers/
2. Write handler functions that receive events
3. Register them at startup, one line per topic

No order service modified. No existing subscriber touched. No testing of unrelated code. This is modularity in practice, not as a principle on a whiteboard, but as a property of the running system.

Why 60 Lines and Not Redis?

OpenAlgo is designed for a single trader running on a laptop or a small VPS. It uses SQLite. It is a single process Python application.

Redis Streams gives you persistence, consumer groups, and multi instance coordination. Kafka gives you distributed processing across data centers. ZeroMQ, which OpenAlgo already uses for market data streaming, gives you cross process messaging.

None of these are needed for a single user platform with three subscribers.

The 60 line event bus does exactly three things:

1. Topic based routing: events go to the right subscribers
2. Async dispatch: subscribers run in a thread pool, never blocking the order response
3. Error isolation: one subscriber crashing does not affect others

When OpenAlgo’s requirements grow, multi user support, multi process deployment, event replay for debugging, the EventBus internals can swap to ZeroMQ or Redis. The event types, the subscribers, and the order services remain unchanged. The right amount of infrastructure today. The right interface for tomorrow.

What This Means for You

If you are a trader using OpenAlgo: your notifications are now reliable. Basket orders send one clean summary. Sandbox mode behaves identically to live mode. Close position always logs. Every order type, every mode, every path, consistent behavior.

If you are building strategies on OpenAlgo’s API: the groundwork is laid for strategy level positions, per strategy risk management, and analytics. These features become possible because the event bus captures what the broker throws away: which strategy owns which order.

If you are a developer contributing to OpenAlgo: the order pipeline is now decoupled. Adding a new consumer is one file and one registration line. The ten order services never need to change for new side effects. And the code is simpler, 15 publish calls replaced 50 plus scattered dispatch points.

If you are building your own trading platform: consider this lesson. The order execution path is the one place where everything connects. Log it, display it, alert it, track it, analyze it, risk manage it. Wire these directly, and you build a system where every new feature requires editing every existing service. Put an event bus in the middle, and you build a system where new features are new files, clean, isolated, and independently testable.

Event driven architecture is not about fancy technology or enterprise patterns. It is about a simple idea: announce what happened, and let everyone who cares decide what to do about it. For trading platforms, where reliability, consistency, and extensibility are not optional, it is the right foundation.

A16z just published a piece that every data practitioner should read. Your Data Agents Need Context makes the case that AI agents have been failing not because the models are bad, but because they’ve been operating without the business context they need to reason well — no understanding of how revenue is defined, which tables are the source of truth, what a fiscal quarter actually means for this organization. The fix, they argue, is a modern context layer: richer than a semantic layer, encompassing canonical entities, tribal knowledge, governance logic, and identity resolution.

They’re right. And I want to extend that argument into territory I think deserves more attention — because the implications of taking it seriously go well beyond the examples that are easiest to reach for.

This Is Data Teams’ Moment — The Question Is How Big We Think

I wrote recently about how agentic AI is finally giving data teams their long-overdue moment. For years, data teams spent budget cycles defending their existence. That is changing fast. AI agents are only as good as the data they run on, and the foundational work data teams have been quietly doing for years — quality, governance, lineage, entity resolution — is now the prerequisite for every enterprise AI initiative that matters.

The budget conversation is flipping — from “justify why we need you” to “how fast can you get us AI-ready?” That’s a genuine shift. But it comes with a risk: if the measure of success for data agents settles at “answering questions BI could already answer,” we’ll have spent enormous energy rebuilding, more expensively and with more failure modes, something that already worked.

Revenue last quarter? Your BI team has a Looker dashboard for that. It refreshes nightly, the fiscal calendar is baked in, and the CFO trusts it. The a16z piece walks through exactly why an agent struggles to replicate even this — wrong revenue definition, stale YAML, ambiguous tables. That’s a real problem worth solving. But it’s also worth asking: once we’ve solved it, what have we actually unlocked?

The answer, I’d argue, is: the foundation for something far more valuable.

The Questions That Context-Plus-Identity Actually Makes Possible

The a16z piece mentions that a modern context layer should go beyond metric definitions to include canonical entities and identity resolution. I want to take that seriously and spell out what it actually unlocks. Walk into any commercial or strategy meeting and listen for the questions that make the room go quiet — not because the answer is sensitive, but because nobody actually knows: How many real customers do we have — not accounts or rows in a CRM, but distinct humans or organisations with an active relationship? Who are our most valuable customers, across product lines, referrals, and renewal behaviour? What is our customers’ lifetime value when “Acme Corp” in the CRM and “ACME Inc.” in billing are finally recognised as the same entity? Who can we upsell, and when, based on what similar cohorts did next? Which customers are three months from churning, based on signals scattered across product, support, and finance systems? These questions couldn’t be asked before not because agents were too dumb, but because the identity layer that makes them answerable was never in place.

What Agents Actually Do with a Resolved Identity Graph

This is where the discussion stops being theoretical. Once entity resolution is in place — once the agent knows who it’s reasoning about across every system — a new class of autonomous workflows becomes possible. Here are four that illustrate the shift.

The Churn Prevention Agent. The agent monitors product usage daily — login frequency, feature adoption, support ticket volume — and when a customer’s pattern matches historical churn signatures, it autonomously cross-references their contract renewal date, checks their NPS history across systems, and drafts a personalised outreach for the customer success manager with a suggested intervention. The entire workflow collapses if the agent is seeing three different records for the same customer across the product database, CRM, and support tool. Entity resolution is not a prerequisite step — it is the agent’s eyes.

The Upsell Timing Agent. A customer’s usage of Product A crosses a threshold. The agent compares their trajectory against cohorts of similar customers who went on to adopt Product B within 90 days, finds the pattern match, creates a qualified opportunity in the CRM, and alerts the account manager — without anyone pulling a report. The cohort comparison is only valid if the purchase history and usage data it’s drawing on are resolved to the same underlying customer. A fragmented identity graph means the agent is comparing apples to a mix of apples and ghosts.

The Cross-Sell Across Household or Org Agent. In B2C — a bank, an insurer, a retailer — one household member has a mortgage, another has a current account, a third just opened a savings product. An agent that understands household relationships, not just individual records, can surface the cross-sell opportunity and route it to the right team before a competitor does. In B2B, the same logic applies at the organisational level: one division is already a customer, another is in a competitor’s trial. The agent identifies the relationship and acts. Both scenarios are structurally impossible without entity resolution at the household or org level — and both are routine once it’s in place.

The Supplier Risk Agent. The agent continuously watches news feeds, financial filings, and logistics data for signals about key suppliers — credit downgrades, port disruptions, leadership changes. When a risk threshold is crossed, it maps which SKUs are affected, what inventory buffer exists, and which alternative suppliers are pre-qualified, then surfaces a recommended action to the procurement team. The catch: this requires knowing that “Acme Logistics Ltd” in the ERP, “ACME Logistics” in the contracts system, and “Acme Log.” in the invoicing platform are the same supplier. Without that, the agent is monitoring fragments, not entities — and the risk signal it’s supposed to catch slips through the gaps.

What “Canonical Entities” Actually Requires

It’s worth dwelling on the practical aspects of having canonical entities and identity resolution as components of a modern context layer — because this is where most implementations will either succeed or quietly fail.

I’ve written before about why building a unified customer identity is genuinely hard, and why so many teams feel like imposters for still wrestling with it. The industry narrative treats unified identity as a prerequisite — something that should already be done before you get to the interesting work. In reality, the vast majority of organisations are still fighting this battle. They’re just not talking about it publicly because it feels like admitting failure.

Entity resolution — sometimes called record linkage, identity resolution, or master data management — is the discipline of determining that two records refer to the same real-world person, company, or object, even when names, formats, and identifiers don’t match. It requires probabilistic matching across messy strings, address normalisation, fuzzy deduplication, graph-based clustering, and ongoing maintenance as records evolve. It’s one of the oldest unsolved problems in enterprise data, not because people haven’t tried, but because the messiness is structural. Organisations grow through acquisition. Customers interact across channels. Nobody standardised data entry in 1997.

For the context layer to include this, entity resolution:

• Cannot be hard-coded into YAML or a semantic layer — it requires ML-powered probabilistic matching trained on actual data

• Cannot be a one-time batch job — it needs to run continuously as new records arrive and existing ones change

• Cannot live outside the warehouse — moving billions of records to a separate resolution system reintroduces the very data silos and lack of context that AI agents are struggling with

• Cannot be fully automated — it needs human-in-the-loop workflows to handle edge cases and feed corrections back into the model

This is not background infrastructure that someone configures once and forgets. It’s an ongoing data capability — closer in character to a production ML system than to a semantic layer definition. Teams that treat it as the former will find their context layers actually working. Teams that treat it as the latter will find their agents confidently wrong.

What This Looks Like When Done Right

Two examples illustrate what the context layer vision looks like when the entity resolution component is actually implemented well.

Fortnum & Mason, the 300-year-old luxury British retailer, had customer data fragmented across restaurant bookings, email signups, online transactions, and in-store purchases. They tried a third-party resolution service first — it created non-persistent identifiers, offered limited visibility, and raised privacy concerns about sending their entire customer dataset externally. By implementing Zingg natively in their Databricks environment, they built persistent, unified customer identifiers across all touchpoints, with full control over the matching process and privacy-compliant resolution that kept sensitive data in their own infrastructure. For the first time in their history, they could understand how customers were shopping across every channel and devise clienteling and personalisation around that. That’s not a BI question answered better. That’s a question that was previously unanswerable and an action that could not be done earlier.

Orthodox Union, operating over 40 websites and 5 mobile applications across disparate CRMs, used Zingg on Snowflake to power their golden records — resolving not just individual identities but household relationships across their entire digital estate. Their agents now reason about constituents as whole people with family connections, not as disconnected records across systems.

These outcomes are exactly what the context layer is promising. They require taking the entity resolution component as seriously as the metric definition component — which means treating it as an engineering discipline, not a checkbox.

The Architecture That Makes This Real

The a16z piece lays out a thoughtful five-step architecture: access the right data, build context automatically, refine with human input, connect to agents, keep it self-updating. That framework is right. The entity resolution layer sits at the foundation of step two — and its quality determines the ceiling for everything above it.

Concretely: before an agent reasons about revenue, lifetime value, churn risk, or upsell opportunity, it needs a stable, persistent entity graph that tells it who “this customer” actually is across every system. That graph needs to be built where the data lives, continuously maintained and grounded in human judgement for edge cases :

Only on top of this identity foundation does the rest of the context layer — the metric definitions, the table routing, the tribal knowledge — become genuinely trustworthy. Knowing what “revenue” means is important. Knowing that the revenue from “Acme Corp” and “ACME Inc.” should be attributed to the same customer is what makes the answer actually right.

The Real Payoff

As I argued in The Data Team’s Moment, AI doesn’t paper over data problems — it runs them at scale, in production, with consequences. A customer success agent operating on a fragmented identity layer doesn’t just give one bad recommendation; it systematically misjudges an entire customer segment, at machine speed, before anyone notices. The autonomous nature of agents is what makes the identity foundation so critical — errors compound rather than get caught.

But the inverse is equally true, and more exciting: an agent operating on a trustworthy identity graph can answer questions that were structurally impossible before. Not because the model got smarter, but because the data it’s reasoning on finally reflects reality.

AI agents that can tell you which accounts are actually the same company, which of those companies are in an expansion window, which customers share a behavioural signature with cohorts that churned six months ago — that’s the payoff the context layer is reaching for. The revenue-growth question is the proof of concept. The customer intelligence actions are the business transformation.

The context layer is necessary. Entity resolution is what makes it sufficient. And the questions worth asking, and the agents worth running — about customers, not metrics — are what make the whole thing worth building.

Building on the a16z post “Your Data Agents Need Context” by Jason Cui and Jennifer Li. Further reading: The Data Team’s Moment and The Identity Crisis. If you’re building the entity resolution layer, Zingg is open source and fully documented.

But a surprising amount of the time is not spent reasoning. It is spent gathering evidence.

You are flipping between your metrics dashboard, log viewer, scheduler, cloud console, SSH sessions, deployment history, and a half-written scratchpad. You are trying to answer questions that are conceptually simple but operationally expensive:

• What changed in the last hour?
• Did the error spike start before or after the deploy?
• Is this one sick node or a fleet-wide regression?
• Did the autoscaling event line up with the HAProxy alert?
• Are we looking at cause, effect, or a noisy side symptom?

Each question requires pulling data from a different tool, in a different format, often with timestamps in different timezones. Then you stitch them into a timeline before you can even begin hypothesizing.

That evidence-gathering phase is where LLMs turned out to be useful for me. Not because they “solve incidents” – the hypothesis formation, the triage, the mitigation calls stay human. But an LLM with access to the right CLIs can chew through the data-gathering faster than I can by hand.

I took the system I’d been building internally and rebuilt it as a package for Pi – a minimal, extensible terminal coding agent – called pi-sre-mode. This post is about the journey from a custom agent platform to a thin extension, and why the value turned out to live in content, not code.

The Predecessor: llmduck#

Before pi-sre-mode, I had a system called llmduck. It went through several rewrites – first in Go, then partially in Rust, and finally as a full TypeScript web application with a server (LLM orchestration, session persistence, WebSocket dispatch), a remote agent (command execution on production hosts with policy enforcement), and a React frontend with a split-panel investigation UI.

The agent had one tool: bash. All infrastructure interaction happened through CLIs – metrics, logs, scheduler state, cloud APIs, SSH. The LLM decided what to run, the agent executed it, and the server tracked the conversation. An SRE methodology skill guided the model through a structured investigation loop: triage, scope, gather evidence across metrics/logs/scheduling, correlate, conclude.

It worked. Give the model good operational guidance and read-only access to the right CLIs, and it can do a lot of the mechanical data gathering that normally burns 20-40 minutes during an incident. It can compare scheduler restarts with metric trends. It can correlate a failing backend with a cloud autoscaling event. It can notice that one host is missing metrics entirely and that the “healthy” cluster is only healthy because the remaining nodes are absorbing the load.

But the runtime was increasingly the wrong abstraction.

I had a custom server, a custom agent, a custom frontend, custom registries for skills and templates, custom session persistence, and a CI pipeline to build and deploy it all. The upstream/wrapper two-repo design – where the generic product lived on GitHub and the org-specific overlay was rsynced into it during build – still meant I was maintaining a bespoke investigation platform.

The more I worked on it, the more obvious it became: the content was the product. The runtime was plumbing.

The skills, the prompts, the methodology, the failure pattern library, the runbooks, the templates – that was where the value lived. Everything else was infrastructure I was maintaining just to get those skills into an LLM session.

Why Pi#

I did not want another chat app. I wanted the investigation to live where I already work: in the terminal, next to the real tools. Pi gave me that, and the specific SDK primitives it exposes turned out to map almost 1:1 to what I had been building by hand in llmduck.

A session looks like this:

/check-connectors
/incident
investigate elevated p99 latency for payments-api, start with a timeline
/report

/incident is an interactive wizard – pick a template, name the service, set a time window.

• /check-connectors – verifies your infrastructure CLIs are reachable
• /report – generates a structured postmortem from the investigation
• /sudo – temporarily bypasses the read-only guardrails

How It Works#

Pi extensions are TypeScript modules that receive an ExtensionAPI object – hooks into the agent lifecycle, slash commands, session state, shell execution, UI primitives, and an inter-extension event bus. The whole extension is one file and the rest is markdown.

The SDK has a few primitives that matter here:

Carrying context across the conversation#

The before_agent_start hook fires before every agent turn. The extension builds a context block from the current incident state and appends it to the system prompt:

pi.on("before_agent_start", (event) => {
  return { systemPrompt: `${event.systemPrompt}\n\n${incidentContext}` };
});

Template, service name, time window, guardrail posture, overlay guidance, preferred skills – all injected automatically. You do not need to keep re-explaining “payments API, last 2 hours, high latency, read-only.” It follows you.

Blocking mutations before they execute#

The tool_call hook intercepts every tool invocation before execution:

pi.on("tool_call", async (event) => {
  const reason = getGuardrailBlockReason(event.toolName, event.input);
  if (reason) return { block: true, reason };
});

The guardrail implementation is regex-based with token boundary detection:

const BLOCKED_PATTERNS = [
  { pattern: /sudo/, reason: "sudo blocked" },
  { pattern: /rm/, reason: "file deletion blocked" },
  { pattern: /kill|pkill|killall/, reason: "process termination blocked" },
  { pattern: /bash\s+-c|sh\s+-c|zsh\s+-c/, reason: "shell trampoline blocked" },
  { pattern: /eval/, reason: "eval blocked" },
  { pattern: /\$\(/, reason: "subshell blocked" },
  { pattern: /systemctl\s+(restart|stop|start)/, reason: "systemctl mutation blocked" },
  { pattern: /aws\s+.*\s+(create|delete|update|put|run|start|stop|terminate|reboot)/,
    reason: "AWS mutation blocked" },
];

The token boundary prevents false positives – remove does not match the rm rule. The write and edit tools are unconditionally blocked. /sudo disables this when needed.

Persisting incident state#

Pi sessions are tree-structured – you can fork, navigate branches, switch contexts. Incident state is persisted as append-only entries:

pi.appendEntry<IncidentModeState>("incident-mode-state", state);

The extension hooks into session_tree and session_fork events to carry incident state across branches automatically. If you fork to explore a side hypothesis, the incident context comes with you.

Commands and UI#

pi.registerCommand() handles slash commands. The /incident wizard uses ctx.ui.select() and ctx.ui.input() for interactive setup – template selection, service name, time window. pi.exec() runs shell commands for connector checks.

Inter-extension events#

Pi has an in-process event bus (pi.events) for cross-package communication – same JS runtime, no network, just pub/sub between extensions loaded from different npm packages. The base package subscribes to a channel, the overlay emits to it at session start. The decoupling is at the import level, not the process level. More on this below.

Templates and Skills#

Most incidents fall into a handful of patterns – 5xx spike, high latency, OOM crash loop, deploy regression, service down. Each pattern has a different starting point: for latency you want percentile breakdowns and dependency response times first; for OOMs you want restart history and memory growth; for a deploy regression you want a before/after comparison.

Templates encode that. When you pick “High Latency” in the /incident wizard, the system prompt gets a focused directive: check p50/p95/p99, look for saturation signals, check upstream dependencies, identify whether queueing is involved. The model starts in the right place instead of flailing. You can also skip the wizard entirely – /incident-5xx payments-api last 2h sets up the investigation directly.

The investigation methodology itself lives in skills – markdown files that get loaded into context:

---
name: sre-methodology
allowed-tools: [Read, Bash, Grep]
---

## Core Loop

1. **Observe**: Gather data. Metrics, logs, scheduler state.
2. **Hypothesize**: Form a theory from the evidence.
3. **Test**: Run a specific query that would confirm or disprove.
4. **Evaluate**: Did the evidence match? Adjust and repeat.

The base package includes two skills: an observe-hypothesize-test-evaluate loop with first principles (evidence over speculation, build timeline before concluding, the loudest symptom is often downstream), and a concrete investigation playbook (scope the blast radius, check recent changes, follow the service path from edge to dependency, narrow using the template focus).

Without the methodology, the model tends to jump to conclusions or fixate on the noisiest symptom. With it, investigations are more systematic.

The Overlay System#

The public package is intentionally generic. The real leverage comes from overlays – private Pi packages that layer org-specific knowledge on top.

An overlay is a separate Pi package that emits an event at session start. The base package picks it up and merges it into the active configuration – templates, connector checks, skills, timezone hints, report output paths:

export default function myOrgOverlay(pi: ExtensionAPI) {
  const overlay: IncidentOverlay = {
    id: "my-org",
    priority: 100,
    timezoneHint: "IST (UTC+5:30); metrics are UTC, logs are IST.",
    reportPathPattern: "rca/{{date}}-{{slug}}.md",
    defaultSkills: ["org-sre-methodology"],
    promptPreamble: "Use org service topology, strict timezone handling...",
    connectorChecks: [ /* ... */ ],
    templates: [ /* ... */ ],
  };

  pi.on("session_start", () => {
    pi.emit("incident-mode:register-overlay", overlay);
  });
}

Templates merge by ID (overlay wins). Skills are deduplicated. Prompt preambles stack across overlays.

What an Overlay Adds#

At work, my private overlay adds investigation templates tied to specific services and a few thousand lines of operational knowledge as skills – how to query our metrics stack, how to read our logs, how to interpret scheduler state, and how to write the postmortem.

The interesting part is what those skills encode. Not just CLI syntax, but the gotchas you learn the hard way: timezone mismatches between tools, CPU metrics that mean something different depending on allocation config, log columns that don’t exist where you’d expect them, query patterns that accidentally match unrelated data. A failure pattern library teaches the model to recognize cascading failures, throughput cliffs (silence often means blocked, not recovered), and the difference between a root cause and a downstream symptom.

This is the kind of institutional knowledge that normally lives in someone’s head or a stale wiki page. Encoding it as skills means it gets applied consistently during every investigation.

Every team has different tools, naming conventions, auth assumptions, and definitions of “normal.” Mix that into the base package and you either leak private knowledge or make the tool unusably abstract. The overlay keeps them cleanly separated.

What the LLM Is Actually Good At#

This is not an “AI will run your ops team” post. The model does not do triage or make mitigation calls. What it does well is the stuff that eats wall-clock time:

Gathering and cross-referencing evidence#

The model does not stop at “good enough.” If you ask it to investigate a latency spike, it will check p50/p95/p99, break it down per node, compare against the pre-deploy baseline, look at upstream dependency latency, check if the error rate moved with it, and then go look at logs for the time window where the percentiles diverged.

Building timelines from messy evidence#

The model is good at stitching together outputs from different tools – logs in IST, metrics in UTC, a deploy at some offset, a scheduler restart in the middle, a cloud instance termination, a queue backup two minutes later – into a coherent narrative. It can produce a first-pass timeline like “memory growth started after deploy X, restart happened 14 minutes later, error rate rose only on one node, autoscaling replaced the sick instance, and downstream proxy alerts were a secondary symptom” faster than I can by hand.

Producing a first draft of the postmortem#

The postmortem is hardest to write right after the incident, precisely when it is most valuable. If the investigation happened inside Pi, /report turns the conversation into a structured markdown RCA draft immediately – timeline, 5 Whys, impact, action items. That alone saves a lot of the “I’ll write the postmortem tomorrow” drift.

Keeping the investigation disciplined#

The methodology skills help the model stay in evidence-first mode: observe before hypothesizing, build a timeline before concluding, distinguish cause from effect, state uncertainty explicitly, say when a connector is missing instead of guessing. Without this, the model tends to fixate on the noisiest symptom. With it, investigations follow an observe-hypothesize-test-evaluate loop.

Where It Still Fails#

Guardrails are not a sandbox#

The read-only protections are operational guardrails, not security boundaries. They catch accidental mutations during investigation, but they are regex patterns – not a sandbox. The real defense is least-privilege at the infrastructure level: an AWS IAM role scoped to read-only, a metrics API token without write access, an SSH key that can only reach jump hosts. If the credentials the agent has cannot mutate anything, it does not matter if a guardrail regex gets bypassed.

The model can still hallucinate#

It will sometimes invent a metric name, over-index on a noisy symptom, or keep running commands when the evidence is already sufficient. Good skills help but do not make it reliable enough to trust unattended.

The Bigger Lesson#

I spent weeks building and rebuilding a custom agent platform. Rewrote it twice in different languages. All plumbing. The things that actually made the investigations good were:

• the SRE methodology skill that encoded how to think about incidents
• the failure pattern library that taught the model to recognize cascade failures and throughput cliffs
• the per-tool skills that encoded gotchas like “that CPU metric means something different in a cgroup” and “silence in the logs means blocked, not recovered”
• the templates that seeded the investigation direction
• the guardrails that kept the model from accidentally mutating production

All of those are markdown files and a few dozen regex patterns. I kept building runtime to deliver content. Once I had a platform that could host markdown skills and intercept tool calls, the custom stack had no reason to exist.

The code is at github.com/mr-karan/pi-sre-mode.

Fin!

ஒரு மரக்கன்று வழங்கும் நிகழ்வில் இருக்கும் மரக்கன்றுகள் என்னை பார்த்து கேட்டன.

சும்மாதானே நிற்கிறாய்! எங்களில் ஒருவரை எடுத்து செல் என்றது.

மரக்கன்றுகளை வளர்க்க மனமிருக்கிறது ஆனால் வீட்டில் இடமில்லை என்று எவ்வாறு அவைகளுக்கு நான் புரியவைப்பேன்! .

There’s a meeting that happens in every organization, usually sometime in Q3 or Q4. A CFO pulls up the budget spreadsheet, squints at the line items, and asks the question that makes every data leader’s stomach drop: “What exactly are we getting out of the data team?”

It’s an uncomfortable question — and for a long time, it didn’t have a clean answer.

Data teams built pipelines nobody asked about, maintained dashboards that were exported to Excel before anyone actually read them, and spent political capital defending headcount to executives who viewed them as a sophisticated cost center. As Joe Reis, author of Fundamentals of Data Engineering, put it plainly: “Data teams will continue struggling to justify their value as long as data in Enterpriseland is a back-office function.”

That was the reality for most of the 2010s and into the early 2020s. Data teams worked in the shadows — vital infrastructure, but rarely celebrated. They were the plumbers of the modern enterprise: essential when things worked, blamed when things didn’t, and invisible the rest of the time.

The era of agentic AI is changing all of that — right now, in real time.

The Old Indignity: Fighting for a Seat at the Table

The data ROI conversation has long been a recurring humiliation dressed up as a business exercise.

Unlike sales, whose numbers are on the board every Monday morning, or marketing, which can point to impressions and conversions, data teams have operated in the murky middle. Their impact is indirect — they help other teams make decisions. As data writer Anna Geller captured it well: “Most data teams work as a support function... their involvement in value creation is indirect. You can’t directly quantify the impact of a new table, dashboard, or pipeline.”

This indirectness has been weaponized against them at budget time. Data leaders find themselves in a paradoxical position: the team responsible for quantification can’t quantify itself. Some resort to building internal P&L reports to justify their own existence. Others try charging other departments “internal fees” for data services — a workaround that says more about the dysfunction of how data is valued than it does about the team’s actual contribution.

Meanwhile, data stacks keep growing enormous. As Monte Carlo’s blog noted, “data teams made history as one of the first departments to spin up 8-figure technology stacks with little to no questions asked” — a fact that makes them conspicuous targets whenever CFOs start scrutinizing every line item with new intensity.

The modern data stack is a marvel of engineering. But without a clear, legible link to revenue, it remains a liability in any budget conversation. At least, it has been — until now.

The Shift Underway: AI Agents Are Looking for a Nervous System

Here’s the thing about agentic AI: it is only as good as the data it runs on.

This seems obvious, but its implications are still rippling through the enterprise in real time. When a company deploys an AI agent to autonomously manage customer service tickets, optimize supply chains, approve procurement requests, or run financial reconciliations, every decision that agent makes is being built on a data foundation. If that foundation is shaky — inconsistent definitions, poor lineage, unresolved duplication — the agent doesn’t just underperform. It fails loudly, in production, with consequences.

Robin Sutara, Field CDO at Databricks, is making this organizational implication explicit in Databricks’ 2025 strategic priorities report: “A successful AI strategy starts with a solid infrastructure. Addressing fundamental components like data unification and governance through one underlying system lets organizations focus their attention on getting use cases into the real-world, where they can actually drive value for the business.”

The work that data teams have been doing for years — data quality, entity resolution, governance, lineage, pipeline reliability — is no longer background noise. It is becoming the prerequisite for the most important initiative in every company.

The budget conversation is flipping. It’s no longer “justify why we need you.” It’s “how fast can you get us AI-ready?”

The Numbers Bearing This Out

The scale of the agentic AI wave is making clear why data teams are starting to matter in a way they simply haven’t before.

Gartner predicts that 40% of enterprise applications will be integrated with task-specific AI agents by the end of 2026 — up from less than 5% today. By 2028, at least 15% of work decisions across organizations will be made autonomously by AI agents, up from virtually zero in 2024. And Gartner’s best-case projection suggests agentic AI could drive approximately 30% of enterprise application software revenue by 2035, surpassing $450 billion.

The old framing — tools automate tasks, people make decisions — is no longer holding. And the people who build, govern, and maintain the data powering this new paradigm are moving from the back office to the top of the boardroom agenda.

The Verizon Test Case

Few leaders are articulating this shift as clearly as Kalyani Sekar, Verizon’s Chief Data Officer. In a 2025 interview with CDO Magazine, she describes the arc of her team’s work — and its growing centrality — with unusual candor:

“Data quality is foundational to Verizon’s long-term goals. As we continue to roll out more analytical use cases, from predictive and prescriptive to generative and agentic, it’s critical that these AI and analytical capabilities, including the reports built on them, are grounded in data that is trustworthy and of the highest quality.”

Verizon handles petabytes of data daily — from network devices, customer touchpoints, service channels. The data team’s job was always to make sense of it and surface insights. Today, that same infrastructure is becoming the operating layer for autonomous AI systems making real-time decisions affecting 140 million wireless retail connections.

The data team isn’t changing. The stakes around its work are.

From Cost Center to Control PlaneFrom Cost Center to Control Plane

McKinsey’s 2025 research on the “agentic organization” is describing what’s unfolding as the largest organizational paradigm shift since the industrial revolution. The key insight for data leaders: in this new world, data quality cannot remain a background maintenance task. It needs to be the foundation every AI agent is built on — before a single workflow goes live.

Nowhere is this more visceral than in entity resolution — the problem of making sure your AI actually knows who it’s dealing with.

I have been watching this play out in real time across industries: “Agentic AI is the race car. Entity resolution is the track. You wouldn’t run a Formula 1 machine on gravel and expect to win.”

The examples are stark. In retail, duplicate customer records lead autonomous agents to dispatch multiple shipments, issue multiple offers, and trigger multiple refunds — quietly eroding margins at scale. In finance, fraudsters opening several accounts with small name variations go undetected because the fraud-detection AI treats each record as a separate person. In healthcare, a patient known as “Mary Chen” at one hospital and “M. Y. Chen” at a clinic becomes two different people in the AI’s world — and an autonomous care coordinator misses her allergy record before booking a procedure.

In each case, the AI wasn’t making bad decisions. It was making confident decisions on bad data. And because agentic systems are autonomous, they don’t make that mistake once — they repeat it hundreds or thousands of times before anyone notices.

Tools like Zingg — which resolve entities at scale across CRMs, EHRs, billing systems, and data lakes, running natively on Snowflake and Databricks — are becoming the kind of foundational data infrastructure that determines whether an AI deployment succeeds or silently fails. One bank that placed Zingg’s entity resolution layer beneath its fraud-detection AI saw fraud detection improve 4x — without changing the AI model at all. The model didn’t get smarter. The data it was reasoning on did.

That’s the new pitch data leaders are making — and it’s a powerful one. AI doesn’t paper over your data problems. It runs them at scale, in production, with consequences. Every dollar not being invested in data quality is a growing tax on every AI initiative downstream.

What This Means for Data Leaders Right Now

The opportunity is real and it is opening — but it requires a change in posture. Data teams have spent years learning to survive by making themselves useful to whoever controls the budget. The new play is positioning as the precondition for everything the business cares about.

Some practical realities for data leaders navigating this moment:

Your backlog is becoming a risk register. Every unresolved data quality issue, every undocumented pipeline, every inconsistent metric definition is a live failure point for an AI agent that will execute on bad information at machine speed. Frame your roadmap accordingly.

Entity resolution is becoming a first-class AI problem. Before an AI agent can act intelligently, it needs to know who it is acting on — and right now, most enterprise data lakes are full of the same customer, supplier, or patient under a dozen slightly different names. Deduplicating and resolving those identities isn’t a one-time cleanup job anymore; it’s an ongoing data foundation that every agentic workflow sits on top of. Teams that are investing in entity resolution infrastructure — using tools like Zingg to continuously resolve identities across CRMs, EHRs, and data platforms — are finding that their AI deployments perform dramatically better, without touching the models at all. The intelligence was always there. The data just wasn’t ready for it.

Data governance is becoming AI governance. The governance frameworks data teams have been building for years — classification rules, data retention policies, lineage tracking — are exactly what enterprises now need to keep AI agents grounded in reality.

The CDO is becoming a critical executive. The function that once reported to the CIO is increasingly driving the AI strategy conversation alongside the CEO and CFO.

The dbt Labs 2025 State of Analytics Engineering survey is already showing the shift: 40% of data teams added headcount in the past year — compared to just 14% the year before. Data budgets are growing too, with 30% of respondents reporting increases versus just 9% in the prior year. The most common complaint is no longer budget cuts — it’s that the team can’t keep pace with the business’s appetite.

As dbt Labs CTO Mark Porter put it directly: “As companies increase AI investments, leaders are prioritizing the teams responsible for data quality and governance — the essential foundation for AI effectiveness.”

The Table Is Turning

There’s something quietly satisfying about watching an industry vindicate itself not through persuasion, but through circumstance.

Data teams have spent years arguing that their work mattered. They built ROI frameworks and internal P&Ls and political coalitions. They embedded with business units and learned to speak in revenue terms. Some of it worked. Most of it was an uphill grind.

Now, agentic AI is making the argument for them — not through rhetoric, but through dependency. The most transformative technology initiative of the decade cannot function without clean data, governed pipelines, and trustworthy infrastructure. And the people building and maintaining those things are, right now, becoming central to the enterprise.

The data team is no longer supporting the business.

The business is starting to run on the data team.

If you found this useful, share it with a data leader who’s still spending Q4 justifying their existence to a skeptical CFO. Their moment is here — and it’s only getting bigger.

The stack that works cleanly for this is ttyd + tmux.

Architecture#

Browser -> Caddy -> ttyd -> nsenter -> su - karan -> tmux(main)

Two decisions matter most:

1. ttyd handles terminal-over-websocket behavior well.
2. -m 1 enforces a single active client, which avoids cross-tab resize contention.

Docker Compose (current)#

services:
  webterm:
    image: tsl0922/ttyd
    container_name: webterm
    restart: unless-stopped
    command: >
      ttyd
        -W
        -p 8080
        -m 1
        nsenter
        -t 1
        -m -u -i -p
        --
        su - karan -c
        "tmux new-session -A -s main"
    privileged: true
    pid: "host"
    networks:
      - public_proxy

Why each flag matters:

• -W: writable shell
• -p 8080: matches my existing Caddy upstream (webterm:8080)
• -m 1: one active client only (no resize fight club)
• nsenter ...: real host shell from inside the container
• su - karan: correct login environment and tmux config loading
• tmux new-session -A -s main: persistent attach/re-attach

Caddy#

terminal.mrkaran.dev reverse proxies to webterm:8080 with TLS via Cloudflare DNS challenge. Because ttyd uses WebSockets heavily, reverse proxy support for upgrades is essential.

tmux profile for agentic workflows#

I tuned tmux for long-running agent sessions, not just manual shell use.

Long-run defaults#

• history-limit 200000
• remain-on-exit on
• window-size latest
• mode-keys vi / status-keys vi

Better operational visibility#

• status line shows host + session + path + time
• pane border shows pane number + current command
• active pane is clearly highlighted

Keybinds I actually use#

Prefix: Ctrl-b

• S: create/attach named session
• N: create named window
• R: rename window
• s: session/window picker
• y: toggle synchronize-panes
• h/j/k/l: pane movement
• H/J/K/L: pane resize

Copy/paste that is not annoying#

This was a big pain point, so I added both workflows:

1. Browser-native copy

   • Ctrl-b m to turn tmux mouse off
   • drag-select + browser copy shortcut
   • Ctrl-b m to turn tmux mouse back on

2. tmux copy mode

   • Ctrl-b [ enters copy mode and shows COPY MODE ON
   • v select, y copy (shows Copied selection)
   • q or Esc exits (shows COPY MODE OFF)

On mobile, ttyd’s top-left menu (special keys) makes prefix navigation workable.

Security model#

This is tailnet-only behind Tailscale. No public exposure.

Still, the container has privileged: true and pid: host, which is a strong trust boundary. If you expose anything like this publicly, add auth in front and treat it as high-risk infrastructure.

Result#

The terminal is now boring in the best way: stable, predictable, and fast to reach from any device.

If someone told me six months ago that anyone could build time series momentum strategies, benchmark them against indices, run multi-asset portfolio backtests, test trend following and trend reversal ideas, or even pair trading strategies — all by just talking to an AI agent — I would have laughed.

I’m not laughing anymore.

I created something called VectorBT Backtesting Skills — a set of AI skills that let you backtest across multiple asset classes (Indian, US, and Crypto markets) with realistic commissions including charges and statutory fees. Within minutes of loading these skills into an AI coding agent, I was running institutional-grade backtests that would have taken hours to code manually. No Python. No debugging. No StackOverflow rabbit holes.

And here’s what really blew my mind, now you can actually have a conversation with your backtesting results. Compare different strategies side by side, ask “how good is this Sharpe ratio?”, understand what the metrics mean, and get a plain-language summary of the pros and cons of your trading strategy. Just talk to your strategy and the AI breaks down every metric for you.

Just plain English instructions like:

“Backtest a simple 10 and 30 EMA crossover on State Bank of India, NSE, daily timeframe, for the last 5 years”

And the AI wrote the code, executed it, fetched real market data, applied realistic transaction costs, benchmarked against NIFTY 50, generated an equity curve, produced a QuantStats tearsheet with 30+ metrics — and then explained the results in plain language so even a beginner could understand them.

Here’s what happened next.

What Are “Skills” and Why Should You Care?

Skills are a new concept introduced by Anthropic and curated by Vercel at https://skills.sh. Think of them as deep domain expertise that you plug into AI coding agents.

Unlike regular prompts or ChatGPT conversations, skills are structured instruction sets that teach an AI agent how to be an expert in a specific domain. They contain best practices, rules, templates, and guidelines that ensure the AI produces high-quality, production-ready output every single time.

The VectorBT Backtesting Skills package (https://github.com/marketcalls/vectorbt-backtesting-skills)

• 5 user-invocable skills — setup, backtest, optimize, quick-stats, strategy-compare
• 12 production-ready strategy templates — EMA Crossover, RSI, Supertrend, MACD, Donchian, and more
• 20 modular rule files — covering everything from transaction costs to walk-forward analysis
• 100+ technical indicators via TA-Lib and OpenAlgo

And it works with 40+ AI coding agents — Claude Code, Cursor, Codex, GitHub Copilot, Gemini CLI, Windsurf, Cline, Roo Code, and more.

The 5-Minute Setup That Changed Everything

Installation is one command:

npx skills add marketcalls/vectorbt-backtesting-skills

Then inside your AI agent:

/setup

That’s it. The AI detects your operating system, creates a Python virtual environment, installs TA-Lib (including the C library), installs VectorBT, Plotly, QuantStats, and 75+ other packages, creates the backtesting folder structure, and configures your API keys.

A setup process that used to take me 30 minutes of googling TA-Lib installation errors — done in one command.

My First Backtest: EMA Crossover on SBIN

I typed:

/backtest simple 10 and 20 ema crossover strategy in SBIN, 
Daily timeframe for the last 5 years.

The AI:

1. Created a backtesting/ema_crossover/ folder
2. Wrote a complete Python script using TA-Lib indicators (not VectorBT built-ins — the skill enforces this)
3. Fetched 5 years of SBIN daily data from OpenAlgo
4. Applied realistic Indian delivery equity fees (0.111% + Rs 20/order) — automatically
5. Cleaned signals with ta.exrem() to remove duplicate entries
6. Ran vbt.Portfolio.from_signals() with proper whole-share sizing
7. Fetched NIFTY 50 as benchmark
8. Printed a comparison table
9. Generated equity curve + underwater plot
10. Exported trades to CSV
11. Created a QuantStats HTML tearsheet
12. Explained the results in plain English

The explanation looked something like this:

* Total Return: Your strategy made 45.23% while NIFTY 50 made 32.10%
  -> BEAT the market by 13.13%
* Max Drawdown: -12.34% — the biggest drop from peak
  -> On Rs 10,00,000 capital, worst temporary loss = Rs 1,23,400
* Sharpe Ratio: 1.45 (return per unit of risk, >1 decent, >2 excellent)

If you’re a beginner, you don’t need to know what a Sharpe ratio is. The AI tells you whether your number is good or bad. If you’re experienced, you get the full QuantStats tearsheet with 30+ institutional metrics.

Where It Gets Crazy: Multi-Asset Portfolio Backtesting

I wanted to test a Supertrend strategy across multiple stocks with custom allocation. Here’s what I typed:

“Backtest a simple Supertrend strategy with parameters 3 and 10 on HDFC Bank, ICICI Bank, State Bank of India, Reliance, Infy, and Tata Steel. Allocation: SBIN 20%, Reliance 30%, Infy 10%, Tata Steel 40%. Daily timeframe, last 5 years. Also add a benchmark column for HDFC Bank fixed deposit at 6.45%.”

The AI built a complete multi-asset portfolio backtest. Individual equity curves for each stock. Combined portfolio performance. Benchmark comparison against NIFTY. And that fixed deposit comparison column I asked for? It added it — showing that my strategy wasn’t even beating a bank FD.

That’s the kind of brutal honesty you need before risking real capital.

The Three Data Routes

Route 1 — Fastest (~10ms)

Historify (DuckDB) → Direct → VectorBT Backtest Engine
~10 ms

Route 2 — Medium (~40–70ms)

Historify (DuckDB) → OpenAlgo API (source=”db”) → Backtest Engine
40–70 ms

Route 3 — Slowest (Seconds to Minutes)

Broker Server → OpenAlgo API (source=”api”) → Backtest Engine
5s — 60s+

Adding Stop-Loss and Take-Profit — In English

After my first backtest, I wanted to add risk management:

“Can you change the backtest with a 5% stop loss?”

Done. New equity curve. New metrics. Drawdown reduced significantly.

“Can you also add a 20% target?

Done again. The AI showed me a before/after comparison — without stop loss vs. with 5% SL and 20% TP. It highlighted that while returns decreased, the drawdown duration dropped drastically.

No parameter documentation hunting. No VectorBT API reference lookup. Just English.

The QuantStats Tearsheet

Every backtest generates an HTML tearsheet via QuantStats. This isn’t a toy — it’s the same kind of report institutional quants produce:

• Cumulative returns vs benchmark
• Rolling Sharpe and Sortino ratios
• Monthly returns heatmap
• Drawdown periods visualization
• Worst 10 drawdowns ranked
• Year-over-year performance breakdown
• Distribution of daily returns
• Monte Carlo simulations
• 30+ risk and return metrics

All generated automatically. No extra code. No extra prompts.

Beyond Backtesting: Dashboards and Indicators

The skills don’t stop at backtesting. There’s a companion Indicator Skills package (https://docs.openalgo.in/skills/indicators) that lets you:

Build Plotly charts with custom indicators:

“Create a Plotly chart of SBIN daily timeframe with Supertrend (3,10), RSI, and MACD”

Build Streamlit dashboards:

“Create a Streamlit dashboard showing sector returns for the last 1 week, 1 month, 3 months, 1 year, 3 years, and YTD for the top 10 NSE sectoral indices, in histogram format”

Replicate TradingView indicators:

I literally copied the name “CM Williams VIX Fix” from TradingView, pasted it into the prompt, and the AI recreated the indicator using OpenAlgo’s 100+ built-in indicators. The values matched exactly.

What Makes This Different from ChatGPT?

I know what you’re thinking. “I can just ask ChatGPT to write backtesting code.”

You can. And here’s what you’ll get:

• Code that uses vbt.MA.run() instead of TA-Lib (wrong approach for serious backtesting)
• Zero fees in the portfolio simulation (makes every strategy look profitable)
• No signal cleaning (duplicate entries that corrupt your results)
• No benchmark comparison
• No QuantStats tearsheet
• Fractional shares in equity backtests (not realistic)
• No plain-language explanation

The skills enforce 20 rules and best practices that prevent these mistakes. Every backtest automatically gets:

• TA-Lib indicators (mandatory — never VectorBT built-ins)
• Signal cleaning with ta.exrem()
• Market-specific realistic fees (Indian delivery: 0.111% + Rs 20, US stocks: ~0.01% + $1, Crypto spot: 0.1%)
• Whole-share sizing (min_size=1, size_granularity=1)
• Benchmark comparison table
• Plain-language explanation
• QuantStats tearsheet

The difference between a ChatGPT backtest and a Skills-powered backtest is the difference between a hobby project and a production-grade analysis.

The Cost Question

Here’s the practical breakdown:

Google Gemini CLI — Free tier available. Best for getting started, running around 5–10 strategies per day, and experimenting without spending anything.

OpenAI Codex — $20/month. Solid execution, reliable for daily use, and a good balance between cost and performance.

Claude Code — $20–200/month. Excellent slash command support, with the $200 plan offering effectively unlimited usage for heavy users.

For a trader just starting out, $20/month on Codex or Claude Code’s base plan gives you more backtesting firepower than most retail quant platforms that charge $100+/month.

The Bigger Picture

This isn’t just about backtesting. This is about what happens when domain expertise becomes installable.

Today it’s backtesting skills. Tomorrow it could be:

• Execution skills — automated straddles, adjustments, portfolio rebalancing
• Risk management skills — real-time position monitoring, Greeks calculation
• Research skills — screening, sector rotation, factor analysis

The VectorBT Backtesting Skills repository is open source. Anyone can contribute. Anyone can create their own skills for their domain.

We’re hitting 100,000 downloads of OpenAlgo next week. The community is building. The skills ecosystem is growing.

The gap between “idea” and “backtest” just collapsed to a single sentence.

Getting Started

# Install the skills
npx skills add marketcalls/vectorbt-backtesting-skills

# Inside your AI agent
/setup
/backtest ema-crossover SBIN NSE D

Three commands. That’s the entire barrier to entry.

Links:

• Repository: https://github.com/marketcalls/vectorbt-backtesting-skills
• Backtesting Skills Docs: https://docs.openalgo.in/skills/backtesting
• Indicator Skills Docs: https://docs.openalgo.in/skills/indicators
• Install OpenAlgo: https://docs.openalgo.in/installation-guidelines/getting-started

Rajandran R is the founder of OpenAlgo, an open-source algorithmic trading platform supporting 30+ Indian brokers. He writes about quantitative trading, AI-assisted development, and market microstructure at marketcalls.in.

A sizeable portion of the Indian and global tech ecosystem was gathered at the India AI Impact Summit in New Delhi last week. While I missed attending it due to personal reasons, as an open source maintainer and an alumnus of The Takshashila Institution's GCPP program, it has been heartwarming to see the summit statement officially mentioning open source - a far cry from the Parisian and the inaugural UK summits that found fairly limited utterances and support. However, I also believe that for these visions of "Sovereign AI" and global trade leadership to succeed, they must be built on a foundation of open, interoperable, and cryptographically verifiable standards.

Over the past year, I've been itching to put some of the skills from the GCPP program to use, but unfortunately, couldn't overcome the initial barrier of shitty first drafts. The opportunity again presented itself this year in the form of the draft Digital Trade Facilitation Bill, 2026, which was announced during the budget in February. Over the last weekend, I formally submitted a technical memorandum with my inputs to the Directorate General of Foreign Trade (DGFT)  regarding this draft.

Below is an executive summary of my recommendations.

Executive Summary

The Digital Trade Facilitation Bill, 2026, is a landmark step toward a paperless, digital-first trade ecosystem (BharatTradeNet). However, to ensure this infrastructure is resilient and inclusive for MSMEs, India must avoid proprietary vendor lock-in.

My submission, therefore, focused on three architectural pillars:

1. Keyless Trust via Transparency Logs: Existing digital signature frameworks often suffer from key management fatigue. I have advocated adopting transparency-log-based signing (modelled on Sigstore). By using short-lived certificates and immutable logs, we can reduce the risk of long-term credential compromise and lower the cost of entry for small exporters.

2. Metadata Integrity & Provenance: A digital Bill of Lading is only as trustworthy as its audit trail. To this end, I've recommended granting legal validity to Software Bills of Materials (SBOMs) and automated provenance records. Utilising Open Container Initiative (OCI) standards ensures that every digital trade artefact has a machine-readable, tamper-proof chain of custody.

3. Ensuring Global Interoperability through Open Standards: To prevent digital islands, I've also recommended that India’s BharatTradeNet must consider utilising globally recognised open source protocols (compliant with UNCITRAL MLETR). This ensures that a digital document issued in Mumbai is natively verifiable by customs and banks in Singapore, Dubai, or Amsterdam without proprietary friction.

Conclusion

Technological sovereignty is not about building walls; it is about building interoperable standards that allow us to compete on a global stage. By anchoring our trade laws in Open Source principles, we ensure that India’s digital backbone is secure, scalable, and sovereign by design.

I have offered to provide a formal technical briefing to the DGFT Technical Committee on these architectures. If you are a founder, architect, or policy-maker navigating these changes, I'd love to hear your thoughts.

Model Format#

Bedrock models use the bedrock/ prefix followed by AWS’s model identifiers:

bedrock/anthropic.claude-opus-4-6-v1
bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0

Nothing surprising here. LiteLLM uses the prefix to route to the right provider.

Authentication#

This is where the first gotcha lives. Bedrock doesn’t use API keys. It authenticates via standard AWS credentials:

• Environment variables: AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY
• AWS profile: AWS_PROFILE
• IAM role (for EC2/ECS/Lambda)

You also need AWS_REGION_NAME set to the region where Bedrock is enabled (e.g., ap-south-1).

The gotcha: don’t pass api_key to LiteLLM when using Bedrock. If you include an api_key parameter in your config, LiteLLM tries to use it instead of the AWS credential chain and auth fails silently. You need to either return None for the API key or omit it from the config entirely.

# Wrong - breaks AWS credential chain
llm_config = {"model": "bedrock/anthropic.claude-opus-4-6-v1", "api_key": some_key}

# Correct - let LiteLLM use AWS credentials
llm_config = {"model": "bedrock/anthropic.claude-opus-4-6-v1"}

This one took a while to figure out because the error messages don’t point you in the right direction.

The top_p and Temperature Conflict#

Bedrock’s Anthropic models reject requests that include both temperature and top_p. If your SDK or framework defaults top_p=1.0, you need to explicitly clear it:

if model.startswith("bedrock/"):
    llm_config["top_p"] = None

Without this, you’ll get a validation error from the Bedrock API. The fix is simple, but the error message isn’t immediately obvious about what’s conflicting.

Inference Profiles#

Bedrock has two types of inference profiles, and the second one is where things get interesting for cost management.

Cross-Region (System-Defined)#

AWS provides these out of the box. They route requests across regions for higher throughput and availability:

global.anthropic.claude-opus-4-6-v1

You can list them with:

aws bedrock list-inference-profiles \
  --type-equals SYSTEM_DEFINED \
  --region ap-south-1

Application Inference Profiles for Cost Tracking#

Application Inference Profiles (AIPs) are tagged wrappers around a model. The killer use case is granular cost attribution via cost allocation tags. Instead of seeing one blob of “Bedrock spend” in your AWS bill, you can break it down by project, team, or service.

Create one with:

aws bedrock create-inference-profile \
  --inference-profile-name my-project-opus-4-6 \
  --model-source "copyFrom=arn:aws:bedrock:ap-south-1::foundation-model/anthropic.claude-opus-4-6-v1:0" \
  --tags key=project,value=my-project \
  --region ap-south-1

This gives you an ARN like:

arn:aws:bedrock:ap-south-1:123456789012:application-inference-profile/abcdef123456

Using AIPs with LiteLLM#

The standard bedrock/ route can’t parse ARNs. Use the bedrock/converse/ route instead:

bedrock/converse/arn:aws:bedrock:ap-south-1:123456789012:application-inference-profile/abcdef123456

The bedrock/ prefix still matches for provider detection and the top_p=None fix, so no code changes needed on your end.

Querying Costs via CLI#

Once your AIPs are tagged, you can query costs using the Cost Explorer API.

Total Bedrock spend for a month:

aws ce get-cost-and-usage \
  --time-period Start=2026-02-01,End=2026-03-01 \
  --granularity MONTHLY \
  --metrics "UnblendedCost" \
  --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Bedrock"]}}' \
  --region us-east-1

Bedrock spend grouped by your cost allocation tag (per-project breakdown):

aws ce get-cost-and-usage \
  --time-period Start=2026-02-01,End=2026-03-01 \
  --granularity DAILY \
  --metrics "UnblendedCost" \
  --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Bedrock"]}}' \
  --group-by Type=TAG,Key=project \
  --region us-east-1

Note: The Cost Explorer API always runs against us-east-1 regardless of where your resources are deployed.

Cost Explorer Setup Checklist#

A few things to get right before cost data starts flowing:

1. The tag must be an active cost allocation tag. Enable it under Billing → Cost Allocation Tags.
2. Use the AIP ARN as the model string in LiteLLM. All invocations through it get tagged automatically.
3. In Cost Explorer, group by tag and select your tag key to see the per-project breakdown.
4. Cost data takes ~24 hours to populate after first usage, so don’t panic if it shows up empty initially.

Dependencies#

LiteLLM needs boto3 to talk to Bedrock:

boto3>=1.28.57

Make sure it’s installed in your environment, otherwise LiteLLM will fail with an import error when you try to use the bedrock/ provider.

Fin!

Which indicator should I use?
Which parameter is best?
Which market works better?

A more important question is often ignored:

How long does my signal actually work?

Because every trading signal has a lifespan.

Not forever.
Not even close.

That lifespan is called the half-life of a signal.

Understanding this one idea can dramatically improve entries, exits, and overall profitability.

Think of a Signal Like Fresh News

Imagine you hear breaking news about a company.

At that exact moment, almost nobody knows it.
You have a real edge.

A few minutes later, more traders see it.
The edge shrinks.

A few hours later, everyone knows.
The edge is mostly gone.

Your information did not become wrong.
It simply became common.

Trading signals work the same way.

A signal is just a piece of information that suggests price is more likely to go up or down.
Over time, other traders react to that same information and push price closer to fair value.

The advantage fades.

What “Half-Life” Really Means

Signal half-life is:

The amount of time it takes for half of a signal’s edge to disappear.

If a signal is strongest right when it appears:

After one half-life
Only half of the original edge remains.

After two half-lives
Only one quarter remains.

After three half-lives
Only one eighth remains.

So even a good signal becomes weak if you wait too long.

This explains why:

Good entries can turn into bad trades.
Good indicators can look useless at the wrong timeframe.
Backtests change dramatically when holding period changes.

Why Traders Lose Money Even With Good Signals

Many traders unknowingly mismatch signal speed and holding time.

Examples:

Using a fast scalping signal but holding trades for hours.
Using a slow trend signal but exiting after ten minutes.

Both situations sabotage the edge.

It is like trying to use milk that expired last week or throwing away fresh bread after one bite.

Same product.
Wrong timing.

Fast Signals vs Slow Signals

Different types of ideas decay at different speeds.

Very Fast Signals

Order flow imbalances
Bid ask pressure
Microstructure patterns

These can stop working in seconds or minutes.

They must be traded quickly.

Medium Speed Signals

Intraday mean reversion
Breakout continuation
Short-term momentum

These usually last minutes to hours.

They fit day trading and short swing trading.

Slow Signals

Multi-week momentum
Value
Carry
Macro themes

These can last weeks, months, or even years.

They require patience and low turnover.

There is no “best” speed.

There is only matching speed.

A Simple Way to Visualize It

Imagine plotting performance versus holding time.

At very short holding times
Results are poor because costs dominate.

At medium holding times
Performance improves and peaks.

At long holding times
Performance falls because the signal has faded.

The peak area is where your signal naturally belongs.

That peak is closely related to half-life.

Why Trading Slower Often Improves Results

Many traders assume faster trading equals more money.

In reality:

Costs increase linearly.
Signal decay happens rapidly at first, then slows.

This means:

Trading too fast bleeds fees and slippage.
Trading too slow captures leftovers.

The sweet spot is usually in between.

When traders lengthen holding periods slightly, they often see:

Higher win rate
Higher Sharpe
Lower stress

Even with the same entry logic.

How You Can Estimate Half-Life Without Complex Math

You do not need advanced statistics.

Try this:

1. Keep the same entry rule.
2. Test multiple exit horizons.
   For example:
   5 bars
   20 bars
   50 bars
   100 bars
3. Record performance metrics for each.
4. Plot or compare results.

Where performance peaks is roughly where your signal’s half-life lives.

That is your natural trading horizon.

A Practical Example

Suppose a breakout strategy shows:

Best results at 30 to 60 minute holds.
Performance drops sharply after 3 hours.
Almost flat after one day.

Conclusion:

This signal is intraday in nature.

Trying to turn it into a swing strategy will not work.
No amount of parameter tuning will fix that.

The Hidden Power of This Concept

Most traders try to improve signals.

Better indicator.
More filters.
More complexity.

Often the bigger improvement comes from:

Using the same signal at the right speed.

You do not need smarter entries.
You need better timing.

Mental Model to Remember

A trading signal is a temporary informational advantage.

Half-life is how fast that advantage evaporates.

Your goal is not to trade as fast as possible.

Your goal is to trade as fast as necessary.

Final Takeaway

If you remember only one thing:

Every signal has an expiration date.
Profitable traders learn when their signal expires and exit before that.

Mastering this concept alone can quietly transform your trading.

One camp says: when you can build a CRM in an afternoon, why pay Salesforce $150 a seat? The other fires back: a POC is not a product. Security, compliance, scalability, ongoing maintenance — the real cost of DIY reveals itself fast.

Both sides have a point. But they’re arguing about the wrong thing.

The real threat to SaaS isn’t that customers will build replacements. It’s that customers will demand their data back.

The New Requirement Nobody Saw Coming

Most enterprises are not going to build their own CRM, their own support desk, or their own payroll system. That’s not happening. But they are going to build custom AI agents — agents tailored to their workflows, their industry, their competitive edge. And those agents are useless without access to the data that lives inside enterprise SaaS systems.

Think about what that means in practice. A procurement agent that can’t read your ERP. A sales agent that can’t query your CRM. A support agent that can’t see your ticketing history. You end up with AI that is powerful in theory and crippled in practice — not because the models aren’t good enough, but because the data is locked behind walls that the enterprise itself cannot breach.

This is the structural tension SaaS has quietly avoided for two decades. Data lock-in was never just a feature — it was the moat. Salesforce, Workday, ServiceNow, SAP built empires not just on software, but on the gravitational pull of accumulated enterprise data. The switching cost wasn’t the product. It was your own history, trapped inside someone else’s system.

AI is now making that lock-in impossible to ignore.

Enterprises building custom agents will demand data portability as a first-class requirement — not a nice-to-have, not a paid add-on, but a baseline expectation. The competitive pressure will shift toward SaaS vendors who open their data, and away from those who treat it as a proprietary asset. MCP is an early signal of this direction, but it is barely the beginning. Real openness means real-time access, structured APIs, granular permissions, and data that can move across systems without friction.

Why We Built Zingg the Way We Did

I want to share something personal here, because it’s directly relevant.

When we built Zingg — an open source entity resolution and data matching tool — one of our earliest and most deliberate architectural decisions was this: Zingg would run natively inside the customer’s own environment. Their data would never move.

Not to our servers. Not to a cloud we managed. Not through a pipeline that touched our infrastructure. Zingg would go to the data, not the other way around.

At the time, this felt like swimming against the tide. The SaaS playbook was clear: centralise everything, host it yourself, lock in the customer through the platform. Everyone was building managed services where customers uploaded their data and got results back. It was convenient. It was scalable. It was the default.

We chose differently — and we took some friction for it. Prospects would ask why they couldn’t just hand us their data and get clean results. Investors would ask why we weren’t building a managed pipeline. The market was optimised for data movement, and we were building for data residency.

Our reasoning was straightforward. Entity resolution deals with some of the most sensitive data an enterprise has — customer records, patient data, financial identities, supplier profiles. The enterprises that needed this most were also the ones with the strictest data governance requirements: banks, healthcare providers, insurers, government agencies. For them, sending data to a third-party system wasn’t a preference issue. It was a compliance wall.

Beyond compliance, there was a deeper principle at work. The enterprise owns its data. The enterprise should control its data. A tool that requires data to leave the enterprise in order to function is not really serving the enterprise — it’s holding the enterprise’s data hostage to deliver a service.

What We Learned From Building This Way

Running natively inside the customer’s environment changed everything about how we thought about the product.

It forced us to build something that was genuinely portable — that could run on on-premise Spark clusters, on Databricks, on AWS EMR, on GCP, on Snowflake, wherever the customer’s data actually lived. It couldn’t be opinionated about infrastructure, because the whole point was to meet the data where it was.

It meant our integration surface was the data itself — structured, well-defined, queryable — not a proprietary pipeline that the customer had to trust and maintain.

And it gave customers something that is genuinely rare in enterprise software: full control and transparency. They could see exactly what Zingg was doing with their data, because Zingg was running on their machines, under their monitoring, in their environment, at their schedule.

There was also a dimension we hadn’t fully appreciated until customers started telling us: it was significantly faster AND cheaper. When your tool runs inside the enterprise’s existing environment, the enterprise leverages infrastructure it has already paid for — compute clusters, data quality frameworks, observability stacks. There is no new data pipeline to design, build, or maintain. There is no ongoing cost of extracting data, shipping it somewhere, and syncing it back. No ETL tax. No vendor markup on compute. The enterprise uses what it already has, and the tool fits around it. For large organisations running entity resolution at scale, this wasn’t a marginal saving — it was a fundamental difference in total cost of ownership.

Interestingly, this also made Zingg stickier in a different way. Not because we had the data — we never did — but because the customer’s confidence in the product grew over time.

Why This Matters Now More Than Ever

Fast forward to today, and the AI agent conversation is making this architectural principle mainstream.

Every enterprise that wants to build custom agents is asking the same question we were thinking about years ago: how do I give my AI access to my data without giving someone else access to my data?

The answer is the same one we arrived at for Zingg. The data should never need to leave.

This is what makes the current MCP momentum so significant. MCP is a protocol that lets AI agents communicate with data systems without requiring data to be extracted, copied, or centralised. It is, in essence, a formalisation of the principle we built Zingg on: bring the compute to the data, not the data to the compute.

But MCP alone is not enough. A faithful MCP implementation means more than exposing an endpoint — it means real-time access, granular permissions, data that reflects the current state of the system, and APIs that are robust enough to support production AI workloads, not just demos.

The enterprises of the next decade will not tolerate data opacity. They have regulators demanding audit trails. They have security teams demanding residency guarantees. And now they have AI ambitions that demand data access. These forces are all pointing in the same direction.

SaaS vendors who open their data thoughtfully — with proper permissioning, auditability, and reliability — will find that openness builds a different kind of loyalty. One that is harder to compete with, because it’s based on trust rather than captivity.

The vendors who resist will find themselves on the wrong side of procurement conversations as AI becomes a standard enterprise requirement. The question won’t be “does your product have AI features?” It will be “can my AI agents use your data?”

If the answer is no, or not without significant friction, the deal will start to look elsewhere.

The Output Was Always the Point

There is one more dimension to this that feels especially relevant now.

Entity resolution is not a peripheral problem. It is one of the most foundational challenges in enterprise data — the question of whether the “Acme Corp” in your CRM is the same as the “Acme Corporation” in your ERP, whether the customer in your support desk is the same person as the one in your billing system, whether your supplier records are duplicated across three different procurement tools. Resolved, unified data is the prerequisite for almost everything else an enterprise wants to do with its data. Analytics, reporting, compliance, machine learning — none of it works well on dirty, fragmented, unresolved data.

This is why we were always deliberate about how Zingg surfaces its output. We didn’t just want Zingg to solve the matching problem — we wanted the resolved data to be genuinely portable, consumable by whatever system or workflow the enterprise chose to use next. Clean, resolved entities written back to the data store of the enterprise’s choice, in formats that fit naturally into existing pipelines, without any proprietary wrapper that would create a new dependency.

We did not see AI agents coming when we made that decision. But in hindsight, it was exactly the right call. A resolved, unified, trustworthy view of an enterprise’s core entities — customers, suppliers, products, locations — is precisely the kind of context foundation that AI agents need to function well. An agent working from fragmented, duplicated, unresolved data will produce fragmented, unreliable answers. An agent working from a clean, resolved data layer can actually be trusted.

We built Zingg to ensure its output could be consumed anywhere the enterprise wanted. It turns out that “anywhere the enterprise wants” now includes AI agents, RAG pipelines, and real-time reasoning systems we couldn’t have imagined when we started. The principle held.

A Principle Worth Standing Behind

When we made the architectural choice for Zingg, we weren’t predicting AI agents. We were just trying to build something enterprises could actually trust with their most sensitive data.

But the principle was right then, and it is right now: data should live where the enterprise decides it lives. Tools — whether they are matching algorithms or AI agents — should go to the data, not the other way around.

The market is catching up to this idea. And those who embrace it, rather than fight it, will be the ones who matter in the next era of enterprise software.

Zingg is an open source entity resolution framework that runs natively in your environment — no data movement required. You can find it at github.com/zinggAI/zingg.

The Core Insight#

Ledger-cli uses a format that is, at its core, just text files. There are no database migrations to manage, no proprietary binary formats, and absolutely no vendor lock-in. The promise is simple: your financial data should outlive the application you use to track it.

Hackability in Gullak’s Context#

Moving to a plain-text format unlocked several advantages that align perfectly with modern AI capabilities and the Unix philosophy.

1. AI-Native Format#

Consider a typical transaction entry in a ledger file:

2026/01/21 Swiggy
    Expenses:Food:Delivery  584.23 INR
    Liabilities:CreditCard:ICICI  -584.23 INR

This structure is trivially parseable by Large Language Models (LLMs). An AI agent can read, write, and reason about these transactions without needing complex serialization logic. Compare this to the friction of extracting data from a SQLite blob or interfacing with a proprietary API like Splitwise’s. The text is the interface.

2. Git-Friendly#

Because every transaction is just a few lines of text, your financial history becomes a git repository. Every change is a diff.

This gives you:

• Full audit trail: git log -p -- main.ledger shows exactly what changed and when.
• Easy rollback: Made a mistake? git revert.
• “What-if” scenarios: Branch off to model a major purchase or a different budget strategy.
• Collaboration: Family budgeting can be handled via Pull Requests.

3. Unix Philosophy#

The file serves as the API. You don’t need export buttons or data liberation requests. You can use standard Unix tools to query your finances.

Find all Swiggy orders over 500 INR:

grep -A2 "Swiggy" main.ledger | grep -E "[5-9][0-9]{2,}|[0-9]{4,}"

Check your monthly food spending:

ledger -f main.ledger bal Expenses:Food -p "this month"

Export to CSV for Google Sheets:

ledger -f main.ledger csv Expenses

4. Extensibility via Comments#

Gullak adds its own metadata using standard ledger comments, which are ignored by the accounting tools but used by the app:

2026/01/21 Zomato
    ; gullak:id 7559a51f
    ; gullak:source whatsapp
    ; gullak:user 919876543210
    Expenses:Food:Delivery  584.23 INR

• gullak:id: A unique ID for CRUD operations.
• gullak:source: Provenance tracking (e.g., entered via WhatsApp, web, or CSV).
• gullak:user: Multi-user support.

Custom tags for your own organization—like ; Recurring: Netflix—just work out of the box.

5. Ecosystem Interoperability#

Because the format is standard, Gullak plays nice with others:

• Paisa: Reads the same file for beautiful visualizations.
• hledger: A Haskell alternative that is drop-in compatible.
• Beancount: Can import ledger files.
• Text Editors: Any editor (VS Code, Vim, Sublime) is a valid client.

What This Enables#

The Trade-off#

Of course, you lose ACID transactions, database indexes, and complex SQL queries. But for personal finance, the scale makes these unnecessary trade-offs.

• You likely have 10-50 transactions a month.
• Running ledger bal on 10 years of data takes less than 100ms.
• The simplicity is the feature.

Conclusion#

Adopting the ledger format turns Gullak from “yet another expense app” into a thin AI layer over your permanent financial record. By decoupling the data from the application logic, we ensure that the data remains accessible, hackable, and enduring.

That equation is changing.

In this article, I want to share how OpenAlgo, combined with a personal AI agent like OpenClaw, fundamentally changes how traders and developers interact with trading systems, APIs, and workflows.

This is not about replacing algorithms or strategies.
It’s about changing the interface between humans and automation.

What Is OpenAlgo?

OpenAlgo is an open-source trading automation platform that provides a unified API layer across 25+ brokers. It allows traders to:

• Place orders programmatically
• Fetch market data and historical data
• Execute complex option strategies
• Schedule and automate trading workflows
• Work with sandbox and live environments

All of this is available without broker lock-in, and without paying for proprietary platforms.

Traditionally, interacting with OpenAlgo meant writing Python code, understanding endpoints, and reading documentation. Powerful — but still technical.

Enter OpenClaw: A Personal AI Automation Agent

OpenClaw is a local, personal AI agent that runs on your machine or server. Unlike chatbots that live only in the browser, OpenClaw can:

• Read local files and codebases
• Execute Python scripts and shell commands
• Access APIs securely
• Maintain long-term memory
• Connect to chat interfaces like Telegram, Discord, WhatsApp, and Slack

Think of it as your own AI operator, not a hosted SaaS chatbot.

When OpenClaw is connected to OpenAlgo, something interesting happens.

You stop calling APIs.
You start talking to your trading system.

From API Calls to Natural Language Trading

Once OpenClaw is given access to OpenAlgo’s API documentation and Python SDK, it builds an internal understanding of:

• Supported order types
• Required parameters
• Symbol formats
• Broker constraints
• Trading workflows

At that point, you can issue instructions like:

• “Get my available funds”
• “Place a CNC order for Reliance, quantity 10”
• “Fetch last 5 days of daily data and save it as CSV”
• “Schedule an order for NIFTY futures at 9:30 AM”
• “Place an ATM straddle and add a 20% stop-loss on both legs”

No endpoint memorization.
No boilerplate scripts.

OpenClaw figures out how to do it by reading the OpenAlgo docs and SDK, then executes the required steps.

Memory Changes Everything

One of the most powerful aspects of OpenClaw is persistent memory.

Once it learns:

• Your preferred order types
• Required strategy fields
• Symbol conventions
• Stop-loss logic
• Scheduling patterns

…it remembers them.

If an order fails due to a missing parameter, OpenClaw learns the correction.
Next time, it doesn’t repeat the mistake.

This transforms automation from “execute commands” into “train an assistant”.

Scheduling, Cron Jobs, and Time-Based Trading

OpenAlgo already supports automation. OpenClaw takes this further by acting as the orchestrator.

You can:

• Schedule trades at exact times
• Run Python scripts via cron jobs
• Execute multi-step workflows:
• Place order
• Fetch positions
• Calculate average price
• Place stop-loss after a delay

All of this can be triggered via natural language from Telegram or WhatsApp.

The execution still happens locally or on your server — not in the cloud — which keeps control and security in your hands.

Options Trading and Strategy Automation

Where this becomes truly powerful is in options trading.

Using OpenClaw with OpenAlgo, you can:

• Place ATM straddles automatically
• Execute iron condors with defined leg distances
• Add percentage-based stop-loss rules
• Calculate break-even points
• Estimate probability of profit
• Fetch and process option Greeks
• Re-enter strategies based on conditions

The agent reads the documentation, figures out the correct API calls, and adapts based on feedback.

This kind of conversational strategy execution is extremely difficult to replicate with traditional rule-based systems.

Historical Data, Backtesting, and Analysis

OpenClaw can also:

• Download historical data via OpenAlgo
• Store it as CSV automatically
• Write Python scripts on the fly
• Run backtests using libraries like VectorBT
• Replace data sources like Yahoo Finance with broker-grade data

You can literally say:

“Backtest a moving average crossover strategy on HDFC Bank for the last 5 years using OpenAlgo data.”

And it will:

1. Fetch the data
2. Write the strategy code
3. Execute it
4. Return the results

At that point, coding becomes optional.

Multi-Channel Control: Telegram, Discord, WhatsApp, Slack

One OpenClaw agent can connect to multiple interfaces at once.

That means:

• Telegram for personal trading control
• Discord for community Q&A
• WhatsApp for server monitoring
• Slack for internal team automation

Each channel can have different permissions.

For example:

• Discord users get read-only access to OpenAlgo documentation
• Personal WhatsApp has full server control
• Telegram handles trading commands only

This makes it possible to run community bots without exposing sensitive systems.

Security and Guardrails Matter

This power comes with responsibility.

OpenClaw can:

• Execute shell commands
• Access files
• Restart services

So guardrails are essential:

• Use read-only permissions for public channels
• Never expose API keys
• Run experiments on isolated machines
• Separate live trading from testing environments

Used carelessly, it can be dangerous.
Used correctly, it’s transformative.

Why This Matters

This isn’t just about trading.

It’s about workflow automation without friction.

Coders gain speed.
Non-coders gain power.
Traders gain leverage.

APIs become conversational.
Automation becomes teachable.
Complexity becomes manageable.

For OpenAlgo users, this means:

• Faster experimentation
• Lower learning curve
• New ways to interact with markets
• Automation without vendor lock-in

Final Thoughts

I’ve largely stopped writing routine code.

Not because coding is obsolete — but because thinking at a higher level is now possible.

Instead of writing functions, I describe intent.
Instead of debugging syntax, I refine logic.
Instead of building tools, I train an assistant.

OpenAlgo provides the foundation.
OpenClaw provides the interface.

Together, they point to a future where automation is conversational, personal, and deeply customizable.

If you’re building something interesting with OpenAlgo and OpenClaw, share it.

This ecosystem is just getting started.

But we might be reinventing the wheel.

The most effective tools for AI agents aren’t those wrapped in heavy “AI-native” integration layers. They are the tools that adhere to a philosophy established forty years ago: the command-line interface.

The Unix Philosophy as an AI Protocol#

An LLM’s native tongue is text. It reasons in tokens, generates strings, and parses patterns. The Unix philosophy, which emphasizes small tools, plain text interfaces, and standard streams, is accidentally the perfect protocol for AI interaction.

Consider the anatomy of a well-behaved CLI:

• Discovery: tool --help explains capabilities without hallucination.
• Structure: tool --json provides deterministic output for parsing.
• Composition: Pipes (|) allow complex workflows to be assembled on the fly.

When you give an agent access to a robust CLI, you don’t need to define 50 separate function schemas. You give it a shell and a single instruction: “Figure it out using --help.”

Context Economy: Lazy vs. Eager Loading#

The current approach to agent tooling often involves dumping massive JSON schemas into the context window. Connecting to a standard MCP server might load dozens of tool definitions, involving thousands of tokens describing every possible parameter, before the user has even asked a question. This is “eager loading,” and it is expensive in terms of both latency and context window utilization.

A CLI-driven approach is “lazy loaded.”

The agent starts with zero knowledge of the tool’s internals. It burns zero tokens on schema definitions. Only when tasked with a specific goal does it invoke man or --help. It retrieves exactly the information needed to construct the command, executes it, and parses the result. This reflects the professional intuition of a senior engineer. We rarely memorize documentation. Instead, we prioritize the ability to quickly discover and apply the specific flags required for the task at hand.

Leveraging the Skills Pattern#

To bridge the gap between a raw CLI and an agent’s reasoning, we can leverage the Skills pattern. This is an emerging standard for agent-based systems where capabilities are documented as self-contained units of knowledge.

Instead of writing a Python wrapper that maps an API to a function call, you provide a Markdown file that explains when and why to use a specific CLI command. The agent uses this as a semantic index.

Here is a snippet from a logchef.md skill:

---
name: logchef
description: Query application logs via LogChef CLI. Use for investigating production incidents and analyzing traffic patterns.
---

## Common Workflows

| Goal           | Command Pattern       |
| -------------- | --------------------- |
| Error Analysis | `logchef sql "..."`   |
| Live Tail      | `logchef query '...'` |

## Example: Error Rates by Minute

To visualize error spikes, use aggregation:

```sql
logchef sql "SELECT toStartOfMinute(_timestamp) as ts, count() as errors
FROM logs.app_logs WHERE service='api-gateway' AND level='ERROR'
GROUP BY ts ORDER BY ts DESC LIMIT 60" --output json
```

When I ask an agent to “check for error spikes in the API gateway,” Claude identifies that this skill is relevant to the request and loads it on-demand. It sees the example, adapts the SQL query to the current context, and executes the CLI command. The Markdown file serves as a few-shot prompt, teaching the model how to use the tool effectively without rigid code constraints.

I maintain similar skill sets for AWS, Kubernetes, and Nomad. The AWS skill doesn’t wrap boto3; it simply documents useful aws ec2 and aws cloudwatch commands.

The Developer Experience: uv and Single-File CLIs#

When a CLI doesn’t exist, the barrier to creating one has never been lower. Modern Python tooling, specifically uv with its inline script metadata, allows us to treat CLIs as disposable, single-file artifacts.

I recently needed an agent to manage my Trello board. Rather than fighting with the Trello API documentation or looking for an abandoned library, I had the agent generate a CLI wrapper:

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = ["typer", "httpx", "rich"]
# ///

import typer
import httpx
import json

app = typer.Typer()

@app.command()
def list_cards(list_id: str, format: str = "table"):
    """Fetch all cards from a specific list."""
    # Implementation details...

This script is self-contained. It defines its own dependencies. It implements --help and --json automatically via typer. It took minutes to generate and immediately unlocked Trello capabilities for the agent.

The SaaS Imperative#

The strategic takeaway for SaaS founders and platform engineers is significant. Your CLI is no longer just a developer convenience; it is your primary AI API.

We are moving past the era where a REST API and a web dashboard are sufficient. If your product lacks a terminal interface, you are locking out the growing workforce of AI agents.

• Browser Automation is brittle, slow, and breaks with every UI update.
• Direct API Integration puts the burden of schema management on the user.
• CLIs offer a stable, discoverable, and composable interface that agents can learn and use autonomously.

The “hobby” CLI wrappers built by enthusiasts, such as those for Notion, Jira, or Spotify, are no longer just developer conveniences. They are becoming critical infrastructure. They provide the stable, text-based interface required for agents to interact with these platforms reliably.

If you want your platform to be AI-ready, don’t just build an MCP server. Build a great CLI. Make sure it supports --json. Write good man pages. The agents will figure out the rest.

“Talk is cheap. Show me the code.” — Linus Torvalds, August 2000

]]> Kailash Nadh https://captnemo.in/blog/2026/01/27/kurbelfahrplan/ https://captnemo.in/blog/2026/01/27/kurbelfahrplan/ Tue, 27 Jan 2026 00:00:00 GMT L3BK7S). We do not use any keys to compact space. Decided to remove the Network Sync code, since I could not parse the fetched content at runtime anyway. Generated a HACKING.md to document the code so i can pick up the session. Fixed a few more silly state bugs. Day 2 Progress I decided to tackle the harder problems, and they were one-shotted fairly nicely: A Devroom configuration page to configure which rooms to show. Create a stands browser Have a home page landing UI. Maps View Saved talks About Page (inheriting the work from the Talk View) The code for all of these features is not very elegant, but I didn’t care that much - it was functional and performant for an app that nobody uses beyond 2 days. Maps I attempted to resize and clean up the maps manually but it was too much effort (Building H map in the app is mine). Then went with Gemini Pro to get a dithered map out instead, which wasn’t perfect either and looked worse at being scaled down. Switched to a “pan view” instead and that worked okay. Still not very happy here3 - I might switch to the maps that are used in the FOSDEM iOS app instead. Overall Notes I’d rate Claude’s contribution at somewhere around 60% - I found it faster to manually fix lots of bugs (also realizing that there is an extra overhead in understanding the code before I can actually fix it). However, things where I don’t have experience - Game state management, input handling - I was able to rely on Claude to get it mostly right. I used Opus throughout the project, and this is the first model where it writes code that is similar in shape to what I’d write. See you at FOSDEM? If you’d like to try the app, come say Hi at FOSDEM. I’ll be at the FOSS United Booth in Building K, Level 2. Or drop me a message. Bug reports are accepted in person only. Kurbelfahrplan is Beerware, so buy me a beer if you like it! I wanted to switch from my custom list view to the gridview in the Playdate SDK and it took me too much time and effort to make the refactor work. ↩ There is no way to get timezone in the playdate lua SDK, so you need a workaround. ↩ One of my ideas is to switch to a leaflet maptile renderer instead and pre-fetch all tiles from https://nav.fosdem.org/. ↩]]> I recently published Kurbelfahrplan, a FOSDEM schedule app for the Playdate. This was the first project where I’ve been quite happy with using Claude Code for development, and used it much beyond Copilot autocomplete and came closer to vibe-coding.

You can see the complete development transcript of my claude-code session. It was interesting as a 2-day greenfield project and I thought it might be nice to document my learnings/process a little bit.

I’d attempted building a game and an app on the playdate before (still WIP) so I had some experience with the tooling. I’d moved on when faced with Lua refactoring challenges¹.

This time around, I had a fairly good idea of what the final app would look like: the playdate has a lot of constraints that restrict the design space, and existing apps all have feature parity.

Here’s some of the things I tried:

Specification

Gave a prompt to Gemini Pro and had it generate a SPEC. It included a link to the SDK, a mention of the timezone workaround ² - I wanted the app to only work in UTC+1 to avoid confusion. I passed it enough context from the schedule.ics file, and on the data storage layer (make separate tables for each Devroom). I don’t have the prompt anymore but it was something like:

write a specification (passed to claude code) for a playdate lua app that will be a fosdem schedule calendar app. See playdate docs at $link. At launch, we want to check the timezone by doing a diff between epochFromTime and epochFromGMTTime, and then download the latest schedule in ICS format from $link. Save the schedule as multiple tables in lua, one table for each category. Create a browser using playdate gridview, display title and location in the first line and a scrollable list of talks below it. Only use the DTSTART, DTEND, SUMMARY, CATEGORIES, URL, LOCATION fields from the ICS file. The user should be able to switch the category with left/right as well as the crank Respect shouldDisplay24HourTime on selecting a talk, show a talk details page with the title, and a QR code for the talk.

I corrected the SPEC a bit, and you can see it (very lightly edited since) in the repo

Day 1

I assumed that Claude wouldn’t have enough training data on Playdate APIs, so I went through the SDK docs, and copied out whatever I felt relevant to a txt file. In my case, I didn’t care for video, audio, and most graphics APIs. This turned out to be 89KB of text. Passed it along with the spec to the Opus in the first session, which immediately got lost in trying to fetch the schedule, so I had to wget it and force it to only read 100 lines. The plan it generated looked okay, so I went ahead.

Fixing Bugs and Progress

The first version was buggy (it did compile), but none of the bugs were major and I got it working in a few more prompts and manual fixes.

1. The timezone code was incorrect, at the first sight.
2. Asked for name suggestions, picked the first one.
3. Ran the app on the simulator, and it broke because it tried to use the network permission incorrectly.
4. A lot of effort on fixing unicode problems (I attempted to solve this at first on the playdate layer, but moved it to python scripts that fetch and parse the schedule).
5. Launched the app on device to realize: Parsing a really large ICS file times out on the device, and generating QR codes takes 15-20 seconds. Switched both to build time. The playdate docs suggest pre-generating QR codes.
6. Lots of layout bugs. In particular, the talk text on the schedule list was never visible and no prompting fixed it - it assumed a black/white color and paint mode bug. I went in and looked and the font height was generated and was too low. This was however expected in the absence of a fast feedback loop.
7. I’d often write one-time scripts with a prompt like:

   Write a Python script inside scripts directory that has zero dependencies,
   fetches https://fosdem.org/2026/schedule/ical, parses the ICS and saves it as
   a large JSON array of arrays. Each item in the large array is a event. The
   event is represented as a list of [ID, SUMMARY,
   CATEGORIES,LOCATION,DTSTART,DTEND] ID is extracted from the URL
   (https://fosdem.org/2026/schedule/event/L3BK7S-free-as-in-burned-out/ ->
   L3BK7S). We do not use any keys to compact space.
   

8. Decided to remove the Network Sync code, since I could not parse the fetched content at runtime anyway.
9. Generated a HACKING.md to document the code so i can pick up the session.
10. Fixed a few more silly state bugs.

Day 2 Progress

I decided to tackle the harder problems, and they were one-shotted fairly nicely:

1. A Devroom configuration page to configure which rooms to show.
2. Create a stands browser
3. Have a home page landing UI.
4. Maps View
5. Saved talks
6. About Page (inheriting the work from the Talk View)

The code for all of these features is not very elegant, but I didn’t care that much - it was functional and performant for an app that nobody uses beyond 2 days.

Maps

I attempted to resize and clean up the maps manually but it was too much effort (Building H map in the app is mine). Then went with Gemini Pro to get a dithered map out instead, which wasn’t perfect either and looked worse at being scaled down. Switched to a “pan view” instead and that worked okay. Still not very happy here³ - I might switch to the maps that are used in the FOSDEM iOS app instead.

Overall Notes

I’d rate Claude’s contribution at somewhere around 60% - I found it faster to manually fix lots of bugs (also realizing that there is an extra overhead in understanding the code before I can actually fix it). However, things where I don’t have experience - Game state management, input handling - I was able to rely on Claude to get it mostly right. I used Opus throughout the project, and this is the first model where it writes code that is similar in shape to what I’d write.

See you at FOSDEM?

If you’d like to try the app, come say Hi at FOSDEM. I’ll be at the FOSS United Booth in Building K, Level 2. Or drop me a message. Bug reports are accepted in person only. Kurbelfahrplan is Beerware, so buy me a beer if you like it!

1. I wanted to switch from my custom list view to the gridview in the Playdate SDK and it took me too much time and effort to make the refactor work. ↩

2. There is no way to get timezone in the playdate lua SDK, so you need a workaround. ↩

3. One of my ideas is to switch to a leaflet maptile renderer instead and pre-fetch all tiles from https://nav.fosdem.org/. ↩

]]> Nemo https://programmerlife1.wordpress.com/2026/01/19/%e0%ae%ae%e0%ae%be%e0%ae%b1%e0%af%8d%e0%ae%b1%e0%ae%99%e0%af%8d%e0%ae%95%e0%ae%b3%e0%af%87-%e0%ae%b5%e0%ae%bf%e0%ae%a9%e0%ae%be-%e0%ae%ae%e0%ae%be%e0%ae%b1%e0%af%8d%e0%ae%b1%e0%ae%99%e0%af%8d/ https://programmerlife1.wordpress.com/2026/01/19/%e0%ae%ae%e0%ae%be%e0%ae%b1%e0%af%8d%e0%ae%b1%e0%ae%99%e0%af%8d%e0%ae%95%e0%ae%b3%e0%af%87-%e0%ae%b5%e0%ae%bf%e0%ae%a9%e0%ae%be-%e0%ae%ae%e0%ae%be%e0%ae%b1%e0%af%8d%e0%ae%b1%e0%ae%99%e0%af%8d/ Mon, 19 Jan 2026 18:56:48 GMT

நீங்கள் ஒரு நாள் சரியாக இல்லாத (messy) code-ஐ கண்டிப்பாக நிச்சயமாக ஏதோ ஒரு வழியில் பெறுவீர்கள்.
அதை எழுதிய developer-ஐ கேலி செய்ய வேண்டாம். சில சமயங்களில் அது நீங்கள் எழுதியதாக கூட இருக்கலாம்.

பல developers ஒரு file-ஐத் திறந்தவுடன்,
“இந்த குப்பையை யார் எழுதியது ? எல்லாத்தையும் rewrite பண்ணிடலாம்” என்று சொல்வதை நான் பார்த்திருக்கிறேன்.

ஆனால் Context மிகவும் முக்கியம்.

அந்த நிர:

• 2 மணி நேரத்தில் delivery செய்ய வேண்டிய சூழலில் எழுதப்பட்டிருக்கலாம்.
• அந்த காலத்தில் hooks, modern tools எதுவுமே இல்லாமல் இருக்கலாம்.
• ஒரே developer, மூன்று பேருடைய வேலையையும் தனியாகச் செய்து கொண்டிருக்கலாம்.

அதனால் code-ஐ மட்டும் பார்த்து தீர்ப்பு சொல்லக் கூடாது.

இதற்கு ஒரு நல்ல விதி இருக்கிறது — Boy Scout Rule.

“நீ கண்ட code-ஐ விட,
அதை கொஞ்சமாவது சுத்தமாக்கி விட்டு போ.”

Day 1-லேயே உலகத்தையே மறு உருவாக்கம்(rewrite) செய்ய வேண்டிய அவசியம் இல்லை.

பல சமயங்களில் அது தேவையில்லாமல் கூட இருக்கலாம்.
வேலை செய்யும் இடங்களில்,
சின்ன சின்ன refactor-களை செய்து,
code-ஐ மெதுவாக நல்ல நிலையில் கொண்டு வருவது தான் புத்திசாலித்தனம் அதுதான் எளிமையாக செய்யக்கூடியதும் கூட.

இன்னொரு முக்கியமான கருத்து — Chesterton’s Fence.

“ஒரு வேலி ஏன் போடப்பட்டது என்று தெரியாமல்,
அதை இடிக்கக் கூடாது.”

அந்த logic, அந்த hack, அந்த workaround —
அது ஏதோ ஒரு காரணத்துக்காக அங்கே இருக்கலாம்.

சில சமயங்களில் காரணங்கள் கூட குறிப்பிடப்பட்டிருக்காது.
அந்த காரணத்தை புரிந்து கொள்ளாமல் மாற்றினால்,
புதிய பிரச்சனைகள் உருவாகும்.

இறுதியாக ஒரு சிந்தனை:

நீங்கள் எதைக் அதிகம் ரசிக்கிறீர்கள்?
இருக்கும் code-ஐ புரிந்து, மெதுவாக மேம்படுத்துவதையா?
அல்லது
புதிதாக, greenfield project-ஐ ஆரம்பிப்பதையா?

இரண்டும் developer வாழ்க்கையின் ஒரு பகுதி தான்.
ஆனால் நல்ல developer என்பவர்,
கெட்ட code-ஐ பார்த்து கோபப்படாமல்,
அதன் பின்னணி கதையை புரிந்து கொண்டு,
அதை நாளுக்கு நாள் சிறப்பாக்குபவரே.

மாற்றங்களே வினா மாற்றங்களே விடை!

ஆகவே சற்று சிந்தித்து செயலாற்றுங்கள்!

]]> Hariharan Uncategorized legacy philosophy programming refactoring https://ravidwivedi.in/posts/brunei/ https://ravidwivedi.in/posts/brunei/ Sat, 17 Jan 2026 17:15:23 GMT In December 2024, Badri and I went to Brunei’s capital, Bandar Seri Begawan. Brunei—officially Brunei Darussalam—is a country in Southeast Asia, located on Borneo Island. It is one of the few remaining absolute monarchies on Earth.

On the morning of the 10th of December 2024, we reached Brunei International Airport by taking a flight from Kuala Lumpur. Upon arrival at the airport, we had to go through immigration, of course. When I was standing in the queue, I was reminded that I hadn’t filled out my arrival card. So I filled it out and submitted it online while I was in the queue.

The immigration officer asked me how much cash I was carrying of each currency. After completing the formalities, the immigration officer stamped my passport and let me in. Take a look at Brunei’s entry stamp in my passport.

We exchanged Singapore dollars to get some Brunei dollars at the airport. The Brunei dollar was pegged 1:1 with the Singapore dollar, meaning 1 Singapore dollar equals 1 Brunei dollar. The exchange rate we received at the airport was the same.

Our (pre-booked) accommodation was located near Gadong Mall. So, we went to the information center at the airport to ask how to get there by public transport. However, the person at the information center told us that they didn’t know the public transport routes and suggested we take a taxi instead.

We came out of the airport and came across an Indian with a bus. The bus seemed more like a minibus by Indian standards. He offered to drop us at our accommodation for 10 Brunei dollars (₹630). As we were tired after a sleepless night, we didn’t negotiate and took the offer. There was nobody else on the bus, and it felt a bit weird using the minibus as our private taxi.

In around half an hour, we reach our accommodation. The place was more like a guest house than a hotel. In addition to the rooms, it had common space consisting of a hall, a kitchen, and a balcony.

Upon reaching the place, we paid for our room in cash, which was 66.70 Singapore dollars (4200 Indian rupees) for two nights. We arrived before the check-in time, so we had to wait for our room to get ready before we entered.

The room had a double bed and also a place to hang clothes. We slept for a few hours before going out at night. We went into Gadong Mall and had coffee at a café named The Coffee Bean & Tea Leaf. The regular caffe latte I had here was 5.20 Brunei dollars. On another note, the snacks we got in Kuala Lumpur covered us for the dinner.

The next day—11th of December 2024—we went to a nearby restaurant named Nadj for lunch. The owner was from Kerala. Here we ordered:

• 1 paneer pepper masala for 5 Brunei dollars (320 rupees)
• 1 nasi goreng pattaya biasa for 4.50 Brunei dollars (290 rupees)
• 1 plain naan for 1.50 Brunei dollars (100 rupees)
• 1 butter naan for 1.80 Brunei dollars (115 rupees)

So, our lunch cost a total of 12.80 Brunei dollars (825 rupees). I didn’t like the fact that the naan was unusually thick.

After the lunch, we planned to visit Brunei’s famous Omar Ali Saifuddien Mosque. However, a minibus driver outside of Gadong Mall told us that the mosque would be closed in half an hour and suggested we visit the nearby Jame’ Asr Hassanil Bolkiah Mosque instead.

He dropped us there for 1 Brunei dollar per person, which seemed like the standard rate for any bus ride in Brunei. The person hailed from Uttar Pradesh and told us about bus routes in Hindi. Bus routes in Brunei were confusing, so the information he gave us was valuable.

It was evening, and we had the impression that the mosque and its premises were closed. However, soon enough, we stumbled across an open gate entering the mosque complex. We walked inside for some time, took pictures, and exited. Walking in Bandar Seri Begawan wasn’t pleasant, though. The pedestrian infrastructure wasn’t good.

Then we walked back to our place and bought some souvenirs. For dinner and breakfast, we bought bread, fruits, and eggs from local shops, as we had a kitchen to cook for ourselves.

The guest house also had a washing machine (free of charge), which we wanted to use. However, they didn’t have detergent. Therefore, we went outside to get some detergent. It was 8 o’clock, and most of the shops were closed already. Others had detergents in large sizes, the ones you would use if you lived there. We ended up getting a small packet at a supermarket.

The next day—the 12th of December—we had a flight to Ho Chi Minh City in Vietnam with a long layover in Kuala Lumpur. We had breakfast in the morning and took a bus to Omar Ali Saifuddien Mosque. The mosque was in prayer session, so it was closed for Muslims. Therefore, we just took pictures from the outside and took a bus to the airport.

When the bus reached near the airport, the bus went straight rather than taking a left turn for the airport. Initially, I thought the bus would just take a turn and come back. However, the bus kept going away from the airport. Confused by this, I asked other passengers if the bus was going to the airport. The driver stopped the bus at Muara Town terminal— 20 km from the airport. At this point, everyone alighted, except for us. The driver went to a nearby restaurant to have lunch.

I felt very uncomfortable stranded in a town that was 20 km from the airport. We had a lot of time, but I was still worried about missing our flight, as I didn’t want to get stuck in Brunei. After waiting for 15 minutes, I went inside the restaurant and reminded the driver that we had a flight in a couple of hours and needed to go to the airport. He said he will leave soon.

When he was done with his lunch, he drove us to the airport. It was incredibly frustrating. On a positive note, we saw countryside in Brunei that we would not have seen otherwise. The bus ride cost us 1 Brunei dollar each.

That’s it for this one. Meet you in the next one. Stay tuned for the Vietnam post!

Thanks to Badri for proofreading.

]]> Ravi Dwivedi https://www.evalapply.org/posts/consciousness-lives-in-a-dedekind-cut/index.html https://www.evalapply.org/posts/consciousness-lives-in-a-dedekind-cut/index.html Fri, 16 Jan 2026 00:00:00 GMT Aditya Athalye meta writing ai intelligence_augmentation tools_for_thought https://ravidwivedi.in/posts/suspension-of-my-riseup-account/ https://ravidwivedi.in/posts/suspension-of-my-riseup-account/ Wed, 07 Jan 2026 18:24:04 GMT Disclaimer: The goal of this post is not to attack Riseup. In fact, I love Riseup and support their work.

Story

Riseup is an email provider, known for its privacy-friendly email service. The service requires an invite from an existing Riseup email user to get an account.

I created my account on Riseup in the year 2020, of course with the help of a friend who invited me. Since then, I have used the email address only occasionally, although it is logged into my Thunderbird all the time.

Fast-forward to the 4th of January 2026, when Thunderbird suddenly told me that it could not log in to my Riseup account. When I tried logging in using their webmail, it said “invalid password”. Finally, I tried logging in to my account on their website, and was told that…

Log in for that account is temporary suspended while we perform maintenance. Please try again later.

At this point, I suspected that the Riseup service itself was facing some issues. I asked a friend who had an account there if the service was up, and they said that it was. The issue seemed to be specific only to my account.

I contacted Riseup support and informed them of the issue. They responded the next day (the 5th of January) saying:

The my-username-redacted account was found inviting another account that violated our terms of use. As a security measure we suspend all related accounts to ToS violations.

(Before we continue, I would like to take a moment and reflect upon how nice it was to receive response from a human rather than an AI bot—a trend that is unfortunately becoming the norm nowadays.)

I didn’t know who violated their ToS, so I asked which account violated their terms. Riseup told me:

username-redacted@riseup.net attempted to create aliases that could be abused to impersonate riseup itself.

I asked a friend whom I invited a month before the incident, and they confirmed that the username belonged to them. When I asked what they did, they told me they tried creating aliases such as floatup and risedown. I also asked Riseup which aliases violated their terms, but their support didn’t answer this.

I explained to the Riseup support that the “impersonation” wasn’t intentional, that the user hadn’t sent any emails, and that I had been a user for more than 5 years and had donated to them in the past.

Furthermore, I suggested that they should block the creation of such aliases if they think the aliases violate their terms, like how email providers typically don’t allow users to create admin@ or abuse@ email addresses.

After I explained myself, Riseup reinstated my account.

Update on the 10th of January 2025: My friend told me that the alias that violated Riseup’s terms was cloudadmin and his account was reinstated on the 7th of January.

Issues with suspension

I have the following issues regarding the way the suspension took place —

• There was no way of challenging the suspension before the action was taken
• The action taken against me was disproportionate. Remember that I didn’t violate any terms. It was allegedly done by a user I invited. They could just block the aliases while continuing the discussion in parallel.
• I was locked out of my account with no way of saving my emails and without any chance to migrate. What if that email address was being used for important stuff such as bank access or train tickets? I know people who use Riseup email for such purposes.
• The violation wasn’t even proven. I wasn’t told which alias violated the terms and how could that be used to impersonate Riseup itself

When I brought up the issue of me getting locked out of my account without a way of downloading my emails or migrating my account, Riseup support responded by saying:

You must understand that we react [by] protecting our service, and therefore we cannot provide notice messages on the affected accounts. We need to act preventing any potential damage to the service that might affect the rest of the users, and that measure is not excessive (think on how abusers/spammers/scammers/etc could trick us and attempt any action before their account is suspended).

This didn’t address my concerns, so let’s move on to the next section.

Room for improvement

Here’s how I think Riseup’s ban policy could be changed while still protecting against spammers and other bad actors:

• Even if Riseup can’t provide notice to blocked accounts, perhaps they can scale back limitations on the inviting account which wasn’t even involved—for example, by temporarily disabling invites from that account until the issue is resolved.

• In this case, the person didn’t impersonate Riseup, so Riseup could have just blocked the aliases and let the user know about it, rather than banning the account outright.

• Riseup should give blocked users access to their existing emails so they have a chance to migrate them to a different provider. (Riseup could disable SMTP and maybe incoming emails but keep IMAP access open). I know people who use Riseup for important things such as bank or train tickets, and a sudden block like this is not a good idea.

• Riseup should factor in the account profile in making these decisions. I had an account on their service for 5 years and I had only created around 5 invites. (I don’t remember the exact number and there’s no way to retrieve this information.) This is not exactly an attacker profile. I feel long-term users like this deserve an explanation for a ban.

I understand Riseup is a community-run service and does not have unlimited resources like big corporations or commercial email providers do. Their actions felt disproportionate to me because I don’t know what issues they face behind the scenes. I hope someone can help to improve the policies, or at least shed light on why they are the way they are.

Signing off now. Meet you in the next one!

Thanks to Badri and Contrapunctus for reviewing this blog post

]]> Ravi Dwivedi https://www.prashanthudupa.com/lifes-purpose/ https://www.prashanthudupa.com/lifes-purpose/ Sun, 04 Jan 2026 18:13:22 GMT Prashanth Udupa Insight Philosophy https://ravidwivedi.in/posts/kuala-lumpur/ https://ravidwivedi.in/posts/kuala-lumpur/ Wed, 31 Dec 2025 14:05:53 GMT In my last post, Badri and I reached Kuala Lumpur - the capital of Malaysia - on the 7th of December 2024. We stayed in Bukit Bintang, the entertainment district of the city. Our accommodation was pre-booked at “Manor by Mingle”, a hostel where I had stayed for a couple of nights in a dormitory room earlier in February 2024.

We paid 4937 rupees (the payment was online, so we paid in Indian rupees) for 3 nights for a private room. From the Terminal Bersepadu Selatan (TBS) bus station, we took the metro to the Plaza Rakyat LRT station, which was around 500 meters from the hostel. Upon arriving at the hostel, we presented our passports at their request, followed by a 20 ringgit (400 rupee) deposit which would be refunded once we returned the room keys at checkout.

Our room was upstairs and it had a bunk bed. I had seen bunk beds in dormitories before, but this was my first time seeing a bunk bed in a private room. The room did not have any toilets, so we had to use shared toilets.

Unusually, the hostel was equipped with a pool. It also had a washing machine with dryers - this was one of the reasons we chose this hostel, because we were traveling light and hadn’t packed too many clothes. The machine and dryer cost 10 ringgits (200 rupees) per use, and we only used it once. The hostel provided complimentary breakfast, which included coffee. Outside of breakfast hours, there was also a paid coffee machine.

During our stay, we visited a gurdwara - a place of worship for Sikhs - which was within walking distance from our hostel. The name of the gurdwara was Gurdwara Sahib Mainduab. However, it wasn’t as lively as I had thought. The gurdwara was locked from the inside, and we had to knock on the gate and call for someone to open it. A man opened the gate and invited us in.

The gurdwara was small, and there was only one other visitor - a man worshipping upstairs. We went upstairs briefly, then settled down on the first floor.

We had some conversations with the person downstairs who kindly made chai for us. They mentioned that the langar (community meal) is organized on every Friday, which was unlike the gurdwaras I have been to where the langar is served every day. We were there for an hour before we left.

We also went to Adyar Ananda Bhavan (a restaurant chain) near our hostel to try the chain in Malaysia. The chain is famous in Southern India and also known by its short name A2B. We ordered

• an onion dosa for 10 ringgits (200 rupees),
• 1 masala tea for 6 ringgits (120 rupees),
• 2 pooris for 8 ringgits (160 rupees) and
• 1 plate potato bajji for 7 ringgits (140 rupees).

All this came down to around 33 ringgits (including taxes), i.e. around 660 rupees. We also purchased some snacks such as murukku from there for our trip.

We had planned a day trip to Malacca, but had to cancel it due to rain. We didn’t do a lot in Kuala Lumpur, and it ended up acting as a transit point for us to other destinations: flights from Kuala Lumpur were cheaper than Singapore, and in one case a flight via Kuala Lumpur was even cheaper than a direct flight!

We paid 15,000 rupees in total for the following three flights:

1. Kuala Lumpur to Brunei,
2. Brunei to Kuala Lumpur, and
3. Kuala Lumpur to Ho Chi Minh City (Vietnam).

These were all AirAsia flights. The cheap tickets, however, did not include any checked-in luggage, and the cabin luggage weight limit was 7 kg. We also bought quite some stuff in Kuala Lumpur and Singapore, leading to an increase in the weight of our luggage.

We estimated that it would be cheaper for us to take only essential items such as clothes, cameras, and laptops, and to leave behind souvenirs and other non-essentials in lockers at the TBS bus stand in Kuala Lumpur, than to pay more for check-in luggage. It would take 140 ringgits for us to add a checked-in bag from Kuala Lumpur to Bandar Seri Begawan and back, while the cost for lockers was 55 ringgits at the rate of 5 ringgits every six hours.

We had seen these lockers when we alighted at the bus stand while coming from Johor Bahru. There might have been lockers in the airport itself as well, which would have been more convenient as we were planning to fly back in soon, but we weren’t sure about finding lockers at the airport and we didn’t want to waste time looking.

We had an early morning flight for Brunei on the 10th of December. We checked out from our hostel on the night of the 9th of December, and left for TBS to take a bus to the airport. We took a metro from the nearest metro station to TBS. Upon reaching there, we put our luggage in the lockers. The lockers were automated and there was no staff there to guide us.

We bought a ticket for the airport bus from a counter at TBS for 26 ringgits for both of us. In order to give us tickets, the person at the counter asked for our passports, and we handed it over to them promptly. Since paying in cash did not provide any extra anonymity, I would advise others to book these buses online.

In Malaysia, you also need a boarding pass for buses. The bus terminal had kiosks for getting these printed, but they were broken and we had to go to a counter to obtain them. The boarding pass mentioned our gate number and other details such as our names and departure time of the bus. The company was Jet Bus.

To go to our boarding gate, we had to scan our boarding pass to let the AFC gates open. Then we went downstairs, leading into the waiting area. It had departure boards listing the bus timings and their respective gates. We boarded our bus around 10 minutes before the departure time - 00:00 hours. It departed at its scheduled time and took 45 minutes to reach KL Airport Terminal 2, where we alighted.

We reached 6 hours before our flight’s departure time of 06:30. We stopped at a convenience store at the airport to have some snacks. Then we weighed our bags at a weighing machine to check whether we were within the weight limit. It turned out that we were.

We went to an AirAsia counter to get our boarding passes. The lady at our counter checked our Brunei visas carefully and looked for any Brunei stamps on the passports to verify whether we had used that visa in the past. However, she didn’t weigh our bags to check whether they were within the limit, and gave us our boarding passes.

We had more than 4 hours to go before our flight. This was the downside of booking an early morning flight - we weren’t able to get a full night’s sleep.

A couple of hours before our flight time, we were hanging around our boarding gate. The place was crowded, so there were no seats available. There were no charging points. There was a Burger King outlet there which had some seating space and charging points. As we were hungry, we ordered two cups of cappuccino coffee (15.9 ringgits) and one large french fries (8.9 ringgits) from Burger King. The total amount was 24 ringgits.

When it was time to board the flight, we went to the waiting area for our boarding gates. Soon, we boarded the plane. It took 2.5 hours to reach the Brunei International Airport in the capital city of Bandar Seri Begawan.

Stay tuned for our experiences in Brunei!

Credits: Thanks to Badri, Benson and Contrapunctus for reviewing the draft.

]]> Ravi Dwivedi https://www.evalapply.org/posts/after-ai/index.html https://www.evalapply.org/posts/after-ai/index.html Mon, 29 Dec 2025 00:00:00 GMT Aditya Athalye meta riff ai intelligence_augmentation tools_for_thought https://openalgo.medium.com/busting-the-common-misconception-faster-languages-alone-dont-guarantee-faster-trade-execution-1fc197a6804c?source=rss-cda86e929c3------2 https://openalgo.medium.com/busting-the-common-misconception-faster-languages-alone-dont-guarantee-faster-trade-execution-1fc197a6804c?source=rss-cda86e929c3------2 Fri, 26 Dec 2025 16:55:51 GMT Busting the Common Misconception: Faster Languages, Desktop Apps, and Broker APIs Do Not Automatically Give You Better Execution or High Frequency Performance

Many traders who get into automation believe a few things that sound convincing at first.

They believe that choosing a faster programming language gives them faster execution.

They believe that building a desktop application on top of a broker API is more reliable than a web based application.

They believe that having broker connectivity over the internet means they can do high frequency trading.

All of these beliefs have some truth behind them, but taken alone they create the wrong picture. To understand what really matters, we need to look at what delays your trades, what high frequency trading actually means, and what kind of tools and SDK support you need to build strategies that can grow over time.

What is latency, explained in simple terms for traders

Latency is the time it takes for your order to travel from your system to your broker, then to the exchange, and for the acknowledgement to come back. Think of it like sending someone a message and waiting for their reply, but in milliseconds instead of seconds. Every time you place an order, this round trip happens.

Latency is not just about your program. It includes the time taken by your internet, your broker network, and the exchange itself.

Where the real delay comes from

Across most broker platforms, the unavoidable round trip between your system and the broker takes anywhere between thirty five and one hundred twenty milliseconds. This is the largest part of the time spent. Your strategy code itself, whether written in Go, Rust, C Sharp, or Python, usually completes its logic in less than one millisecond.

OpenAlgo adds around five milliseconds because it takes care of routing, caching, and consistent internal checks before your order is sent out.

Put together, the total time looks like this:

Strategy code time         around 0.1 ms
OpenAlgo internal time     around 5 ms
Broker request time        35 to 120 ms
Total                      40 to 125 ms

Removing OpenAlgo would save a few milliseconds, but it would not remove the biggest delay, which is the broker path.

Why many traders cannot do true high frequency trading from home

Many new traders say that a total latency of forty to one hundred twenty milliseconds means they can do high frequency trading. In reality, true high frequency trading takes place at microsecond and nanosecond timings. These strategies do not use the regular internet. They operate inside exchange data centers where the cables between systems are measured in meters and sometimes even centimeters.

Over home or office internet, latency is not only higher, it is also inconsistent. One order might reach the broker quickly, the next one might take longer because of changing network conditions. For high frequency strategies, consistency matters even more than speed. To get consistent speed, traders colocate their systems inside exchange facilities. That is where true high frequency strategies live. They do not run on a laptop or a desktop at home, and they do not run over regular internet.

So the myth that high frequency can be done over a standard broker API with internet based round trips is not accurate. To get consistency, you need physical proximity to the exchange and infrastructure designed for it.

The myth of desktop applications being more reliable than web applications

Another common belief is that desktop based applications built on top of broker APIs are more robust than web applications. Traders often assume that a desktop application is closer to the hardware and therefore more stable.

In reality, reliability comes from connection handling, retry logic, persistent sessions, consistent state management, and good error recovery. These can be implemented in both desktop and web applications. A desktop wrapper on top of an unstable API does not magically make that API more stable. A well built web application with proper retry, reconnect, and session persistence can be more reliable than a desktop application that does not handle failures correctly.

The difference is not the interface you see. The difference is the engineering underneath.

Why SDK support matters more than many traders realize

Most small and mid sized brokers provide only a single SDK, usually in Python. Some brokers do not provide any SDK at all, leaving you to write everything yourself. If you want to build strategies in Go, Rust, Java, C Sharp, or Node.js, you often have to start from raw HTTP requests.

OpenAlgo offers ready to use SDKs in all of these languages.
Python, Node.js, Java, Rust, DotNet using C Sharp, and Go.

This means you can write strategies in the language you prefer and target multiple brokers without building integrations from scratch. That capability saves time and allows you to move between brokers without rewriting your entire system.

A few milliseconds of overhead is small compared to the time saved when you do not need to rebuild your trading infrastructure every time you switch providers.

So what is the truth traders should know

Language speed helps, but does not break the biggest delay
Desktop applications are not automatically more robust
The internet is not consistent enough for true high frequency trading
Broker round trip latency is measured in tens of milliseconds
Actual high frequency trading is measured in microseconds and nanoseconds

Consistency requires colocating near exchanges, not sitting behind a home connection

And most importantly A multi language SDK ecosystem gives more real world advantages than shaving a few milliseconds off local processing

The misconception resolved

Speed matters, but it is not where most traders think it is.
Consistency matters, but you do not get it over the public internet.
Robustness matters, but it is a property of system design, not the application type.

And flexibility matters, because strategies grow over years, not days.

The right foundation makes your trading platform durable.
OpenAlgo gives that foundation through consistent broker access and ready to use SDKs in the languages traders want to build with, which many brokers do not offer at all.

]]> Rajandran R (Creator - OpenAlgo) algorithmic-trading kit-sdk trading openalgo python https://mrkaran.dev/posts/logchef-v1/ https://mrkaran.dev/posts/logchef-v1/ Mon, 22 Dec 2025 00:00:00 GMT 100”), pick a frequency, configure severity and labels. Logchef runs your query on schedule, evaluates the threshold, and if it triggers, fires an alert. Alert history is stored with execution logs so you can debug why something fired (or didn’t). LogchefQL: From Toy Parser to Production Backend# The query language I wrote about originally was pretty basic – just filters that compiled to SQL on the frontend. Over the months it grew into something more capable, but more importantly, I rewrote the entire parser in Go and moved it to the backend. This also opens the door for a CLI tool later – same parser, same query language, different interface. Here’s what LogchefQL looks like now: namespace="prod" AND level="error" | message, trace_id The pipe operator (|) selects specific columns instead of SELECT *: msg.level="ERROR" | timestamp, msg.request.method Dot notation handles nested JSON fields. If your logs have a log_attributes Map column with nested data: log_attributes.user.name = "john" For keys that contain dots (common in OTEL-style logs), use quoted field syntax: log_attributes."http.status_code" >= 500 Why Move Parsing to the Backend?# The original frontend parser was TypeScript. It worked, but had problems: Inconsistency: The frontend generated SQL, but the backend had no idea what that SQL meant. Validation happened in two places. Type-awareness: ClickHouse has Map, JSON, LowCardinality, and various string types. The frontend didn’t know the schema, so it couldn’t generate optimal SQL for each column type. For a Map(String, String) column, you want mapContains() or ['key'] access. For JSON, you want JSONExtractString(). For regular String, it’s a simple comparison. Debugging hell: When a query failed, was it the parser? The SQL generator? ClickHouse syntax? Everything happened client-side, invisible to server logs. The new architecture is cleaner: Frontend Backend | | | --- LogchefQL query --> | | | --> Parse (Go) | | --> Validate against schema | | --> Generate type-aware SQL | <-- SQL + results ----- | --> Execute on ClickHouse The backend exposes three endpoints: /logchefql/translate (returns the SQL for “View as SQL”), /logchefql/validate (real-time validation with debouncing), and /logchefql/query (parse, validate, execute, return results). Moving parsing to the backend also made the field sidebar implementation cleaner – the same schema-aware code that generates WHERE clauses can filter field values based on your current query. The Field Sidebar# If you’ve used Kibana, you know the interaction: click a field, see its top values, click a value to add it as a filter. It’s the fastest way to explore logs when you don’t know exactly what you’re looking for. Building this for ClickHouse required solving a few problems: High-Cardinality Fields# You can’t just run SELECT DISTINCT field FROM logs on a table with billions of rows. String fields like trace_id would take forever and return millions of values. The solution is a hybrid loading strategy based on column types: LowCardinality and Enum fields: Auto-load values when the sidebar opens. These are designed for fields with limited distinct values. String fields: Require an explicit click. A badge shows the count is unknown until you ask. Complex types (Map, Array, Tuple, JSON): Excluded. You can’t have meaningful “distinct values” for a JSON blob. Progressive Loading# Each field loads in parallel (max 4 concurrent) with a 15-second timeout. One slow or failed field doesn’t block others – you get a retry button for that specific field. Query Context# The sidebar respects your current query. If you’ve filtered to level="error", the field values update to show only values from error logs. This happens through the backend – the field values endpoint accepts the current LogchefQL query and applies it as a WHERE clause filter. Same parser, same SQL generator, consistent results. Query Cancellation# Hit Esc and it cancels the query in ClickHouse. Without this, pressing “Cancel” would just hide the spinner – the query kept running on the server, burning resources. The implementation uses ClickHouse’s query ID feature: SELECT * FROM logs WHERE ... SETTINGS query_id = 'logchef-abc123' When you hit Esc, the frontend calls a cancellation endpoint that runs: KILL QUERY WHERE query_id = 'logchef-abc123' The original query returns an error, the UI clears, ClickHouse frees resources. Simple, but requires plumbing the query ID through every execution path. AI Query Assistant# “Write a query that finds slowest endpoints by p99” actually works. The AI generates LogchefQL or SQL based on natural language and your table schema. Under the hood it uses go-openai, so any OpenAI-compatible endpoint works – OpenAI, Ollama, vLLM, whatever you prefer. The system prompt includes your table schema so the model knows what fields exist. There’s also an MCP server that exposes Logchef to AI assistants like Claude Desktop, Cursor, or any MCP-compatible client. Instead of context-switching between your AI chat and the log viewer, you can ask directly: “What log sources do I have access to?” “Find all 500 errors in the last hour from the web service” “Show me a histogram of log volume over the past day” “What are the most common error messages in the database logs?” The MCP server handles discovery (teams, sources, schemas), querying (full ClickHouse SQL), analysis (histograms, saved queries), and even admin operations. It’s a separate binary that runs alongside Logchef – configure it once, and your AI assistant can query your logs through natural conversation. Compact View for Terminal Lovers# Not everyone wants a table. The compact view is a terminal-style display that shows logs as formatted text with syntax highlighting. Denser and faster to scan for certain debugging workflows. Query Variables# Use {{namespace}} in your query, and an input field appears automatically. Great for saved queries that teams want to reuse with different parameters. This was a community contribution from @songxuanqing. The implementation detects {{variable}} patterns in the query text and renders input fields dynamically. Team Management and RBAC# Logchef supports multi-tenancy with role-based access. Teams can have multiple data sources, and users can be members of multiple teams with different roles: Admin: Full access, can manage team members and sources Editor: Can create/edit saved queries and collections Viewer: Read-only access to query and explore logs This integrates with OIDC for SSO, so you can use your existing identity provider. Admin UI for Runtime Config# Configure stuff without touching config files. The admin settings panel lets you change AI configuration, Alertmanager connection, authentication settings, and query timeouts. This was a migration from config files to database-backed settings. On first boot, Logchef seeds the database from config.toml. After that, the UI takes over and changes are stored in SQLite. Backward compatible – existing config files still work, the UI just overrides them at runtime. No more SSH-ing into production to bump a timeout. Prometheus Metrics# A /metrics endpoint exposes query execution times, error rates, active queries, and other operational data. There’s a pre-built Grafana dashboard for monitoring Logchef itself. What’s Not in 1.0# Some things didn’t make the cut: Live tail: Streaming logs in real-time. Still on the roadmap. Dashboarding: Multiple visualizations on one page. Logchef is query-focused; for dashboards, you probably want Grafana with ClickHouse as a datasource. Calling It 1.0# Calling something “1.0” is weird. There’s no clear line where software becomes “ready.” But I’ve been using Logchef daily at work for months now, and it’s at the point where I trust it. The rough edges are mostly smoothed out. The architecture feels right. Building tools you use yourself is different. You’re the first to hit the rough edges, so you fix them. Slower than building for imaginary users, but the result is something you actually want to use. Thanks again to Kailash for the early direction (schema-agnostic was his idea), and to everyone at Zerodha who’s been using this and giving feedback. Thanks to @songxuanqing for query variables and other contributors for docs and bug fixes. Demo | Docs | GitHub | v1.0.0 Release Fin!]]> About eight months ago I wrote about Logchef – a log viewer I’d been building to scratch my own itch with log exploration at work. Back then it was basically a nicer way to query ClickHouse without writing raw SQL every time. Today I’m shipping v1.0, and it’s evolved into something I didn’t quite expect.

Let me walk through the major features that made it to 1.0 and some of the engineering decisions behind them.

Alerting with Alertmanager Integration#

In that first post, I mentioned alerting as a “roadmap” item. It always felt like the obvious next step – you find a pattern in your logs, you want to know when it happens again.

But building it took longer than expected. My first attempt was a “rooms” system – a home-grown notification router with its own email, Slack, and webhook channels. I got it working, then stared at the code for notification deduplication, grouping, silencing, and escalation. All problems that Alertmanager has already solved and battle-tested in production for years.

So I ripped out rooms and integrated Alertmanager instead. Now Logchef just fires alerts to Alertmanager, and you get all the routing logic – Slack, PagerDuty, email, webhooks, silencing, grouping, inhibition – without me reinventing it poorly.

The workflow is simple: write a LogchefQL or SQL query, set a threshold (e.g., “fire if count > 100”), pick a frequency, configure severity and labels. Logchef runs your query on schedule, evaluates the threshold, and if it triggers, fires an alert. Alert history is stored with execution logs so you can debug why something fired (or didn’t).

LogchefQL: From Toy Parser to Production Backend#

The query language I wrote about originally was pretty basic – just filters that compiled to SQL on the frontend. Over the months it grew into something more capable, but more importantly, I rewrote the entire parser in Go and moved it to the backend. This also opens the door for a CLI tool later – same parser, same query language, different interface.

Here’s what LogchefQL looks like now:

namespace="prod" AND level="error" | message, trace_id

The pipe operator (|) selects specific columns instead of SELECT *:

msg.level="ERROR" | timestamp, msg.request.method

Dot notation handles nested JSON fields. If your logs have a log_attributes Map column with nested data:

log_attributes.user.name = "john"

For keys that contain dots (common in OTEL-style logs), use quoted field syntax:

log_attributes."http.status_code" >= 500

Why Move Parsing to the Backend?#

The original frontend parser was TypeScript. It worked, but had problems:

1. Inconsistency: The frontend generated SQL, but the backend had no idea what that SQL meant. Validation happened in two places.

2. Type-awareness: ClickHouse has Map, JSON, LowCardinality, and various string types. The frontend didn’t know the schema, so it couldn’t generate optimal SQL for each column type. For a Map(String, String) column, you want mapContains() or ['key'] access. For JSON, you want JSONExtractString(). For regular String, it’s a simple comparison.

3. Debugging hell: When a query failed, was it the parser? The SQL generator? ClickHouse syntax? Everything happened client-side, invisible to server logs.

The new architecture is cleaner:

Frontend                  Backend
   |                         |
   | --- LogchefQL query --> |
   |                         | --> Parse (Go)
   |                         | --> Validate against schema
   |                         | --> Generate type-aware SQL
   | <-- SQL + results ----- | --> Execute on ClickHouse

The backend exposes three endpoints: /logchefql/translate (returns the SQL for “View as SQL”), /logchefql/validate (real-time validation with debouncing), and /logchefql/query (parse, validate, execute, return results).

Moving parsing to the backend also made the field sidebar implementation cleaner – the same schema-aware code that generates WHERE clauses can filter field values based on your current query.

The Field Sidebar#

If you’ve used Kibana, you know the interaction: click a field, see its top values, click a value to add it as a filter. It’s the fastest way to explore logs when you don’t know exactly what you’re looking for.

Building this for ClickHouse required solving a few problems:

High-Cardinality Fields#

You can’t just run SELECT DISTINCT field FROM logs on a table with billions of rows. String fields like trace_id would take forever and return millions of values.

The solution is a hybrid loading strategy based on column types:

• LowCardinality and Enum fields: Auto-load values when the sidebar opens. These are designed for fields with limited distinct values.
• String fields: Require an explicit click. A badge shows the count is unknown until you ask.
• Complex types (Map, Array, Tuple, JSON): Excluded. You can’t have meaningful “distinct values” for a JSON blob.

Progressive Loading#

Each field loads in parallel (max 4 concurrent) with a 15-second timeout. One slow or failed field doesn’t block others – you get a retry button for that specific field.

Query Context#

The sidebar respects your current query. If you’ve filtered to level="error", the field values update to show only values from error logs. This happens through the backend – the field values endpoint accepts the current LogchefQL query and applies it as a WHERE clause filter. Same parser, same SQL generator, consistent results.

Query Cancellation#

Hit Esc and it cancels the query in ClickHouse. Without this, pressing “Cancel” would just hide the spinner – the query kept running on the server, burning resources.

The implementation uses ClickHouse’s query ID feature:

SELECT * FROM logs WHERE ...
SETTINGS query_id = 'logchef-abc123'

When you hit Esc, the frontend calls a cancellation endpoint that runs:

KILL QUERY WHERE query_id = 'logchef-abc123'

The original query returns an error, the UI clears, ClickHouse frees resources. Simple, but requires plumbing the query ID through every execution path.

AI Query Assistant#

“Write a query that finds slowest endpoints by p99” actually works. The AI generates LogchefQL or SQL based on natural language and your table schema.

Under the hood it uses go-openai, so any OpenAI-compatible endpoint works – OpenAI, Ollama, vLLM, whatever you prefer. The system prompt includes your table schema so the model knows what fields exist.

There’s also an MCP server that exposes Logchef to AI assistants like Claude Desktop, Cursor, or any MCP-compatible client. Instead of context-switching between your AI chat and the log viewer, you can ask directly:

• “What log sources do I have access to?”
• “Find all 500 errors in the last hour from the web service”
• “Show me a histogram of log volume over the past day”
• “What are the most common error messages in the database logs?”

The MCP server handles discovery (teams, sources, schemas), querying (full ClickHouse SQL), analysis (histograms, saved queries), and even admin operations. It’s a separate binary that runs alongside Logchef – configure it once, and your AI assistant can query your logs through natural conversation.

Compact View for Terminal Lovers#

Not everyone wants a table. The compact view is a terminal-style display that shows logs as formatted text with syntax highlighting. Denser and faster to scan for certain debugging workflows.

Query Variables#

Use {{namespace}} in your query, and an input field appears automatically. Great for saved queries that teams want to reuse with different parameters.

This was a community contribution from @songxuanqing. The implementation detects {{variable}} patterns in the query text and renders input fields dynamically.

Team Management and RBAC#

Logchef supports multi-tenancy with role-based access. Teams can have multiple data sources, and users can be members of multiple teams with different roles:

• Admin: Full access, can manage team members and sources
• Editor: Can create/edit saved queries and collections
• Viewer: Read-only access to query and explore logs

This integrates with OIDC for SSO, so you can use your existing identity provider.

Admin UI for Runtime Config#

Configure stuff without touching config files. The admin settings panel lets you change AI configuration, Alertmanager connection, authentication settings, and query timeouts.

This was a migration from config files to database-backed settings. On first boot, Logchef seeds the database from config.toml. After that, the UI takes over and changes are stored in SQLite. Backward compatible – existing config files still work, the UI just overrides them at runtime. No more SSH-ing into production to bump a timeout.

Prometheus Metrics#

A /metrics endpoint exposes query execution times, error rates, active queries, and other operational data. There’s a pre-built Grafana dashboard for monitoring Logchef itself.

What’s Not in 1.0#

Some things didn’t make the cut:

• Live tail: Streaming logs in real-time. Still on the roadmap.
• Dashboarding: Multiple visualizations on one page. Logchef is query-focused; for dashboards, you probably want Grafana with ClickHouse as a datasource.

Calling It 1.0#

Calling something “1.0” is weird. There’s no clear line where software becomes “ready.” But I’ve been using Logchef daily at work for months now, and it’s at the point where I trust it. The rough edges are mostly smoothed out. The architecture feels right.

Building tools you use yourself is different. You’re the first to hit the rough edges, so you fix them. Slower than building for imaginary users, but the result is something you actually want to use.

Thanks again to Kailash for the early direction (schema-agnostic was his idea), and to everyone at Zerodha who’s been using this and giving feedback. Thanks to @songxuanqing for query variables and other contributors for docs and bug fixes.

Demo | Docs | GitHub | v1.0.0 Release

Fin!

]]> Karan Sharma https://captnemo.in/blog/2025/12/12/berlin/ https://captnemo.in/blog/2025/12/12/berlin/ Fri, 12 Dec 2025 00:00:00 GMT Haven’t posted here in a while, but this is worth an update: I’m moving to Berlin. I’ve loved living in Bangalore for the last decade (I moved here just after the first HillHacks in May 2015), but it is time for an adventure.

For the last couple of years, I’ve spent my time living in Indiranagar, building communities, helping underline.center, working on blr.today, organizing events, and really doing things that I care about. It has been a wonderful time, but we wanted to experience life elsewhere, and Berlin seems to fit the bill.

Bangalore’s infrastructure has been in the news, and it is a part of why I’m moving. An unwalkable footpath on CMH road meant we spent half a year dealing with a broken elbow this year. At Takshashila, I was taught that I’m not allowed to bring up civic problems without also coming up with solutions. But Bangalore is an unsolvable paradox: A city with 50+ unicorns and no walkable footpaths.

I’ve fought my personal share of battles against the state, but this is one where I don’t have any hope of making a difference. I appreciate the work that the Bangalore Civil Society is doing in attempting to hold the missing-government accountable - it just isn’t the kind of work that I want to do. Fighting for basic necessities (clean air, walkable footpaths, open public spaces, well-funded public transit) shouldn't be anyone's job in a city as large as Bangalore. I’m picking Exit (for now).

Why Berlin? Mainly because I have lots of friends there. Our immigration journey is still quite early, so maybe I can write about it when things have stabilized a bit. For now, if you have any Berlin recommendations or connections, please send them my way.

]]> Nemo https://mrkaran.dev/posts/fixing-cibil-with-ai/ https://mrkaran.dev/posts/fixing-cibil-with-ai/ Thu, 04 Dec 2025 00:00:00 GMT About a month ago, I downloaded my CIBIL report expecting a routine check. Instead, I found loans from lenders I had never interacted with, written-off accounts, overdues from fintechs I had never installed, and even two-wheeler loan enquiries. I don’t even ride a bike.

My credit score had collapsed to under 680. I stared at the report trying to understand how this could happen.

The Root Cause: A Wrong Date of Birth#

Buried in my profile section was the problem: my date of birth was wrong. Not a typo, but a completely different year.

Because of this mismatch, CIBIL’s system had paired my PAN and mobile number with someone else’s DOB, effectively merging two individuals’ credit histories into one report. The accounts mapped to me included:

• Aditya Birla Capital: Short-term personal loan marked as doubtful/substandard
• Clix Capital: A loan marked written-off (₹50,000+)
• Poonawalla Fincorp: Personal loans with delayed payments
• Ring / Kissht: Unsecured digital loans
• InCred: Personal loan I never took
• Dhani Loans: BNPL-style loan with unrecognized activity
• Axio (Capital Float): Old consumer loan
• KrazyBee: Various short-term loans
• Transactree: Small-ticket personal loan
• Multiple enquiries from HDFC, ICICI, IDFC First, Shriram Finance, and others

Some were written-off, others 90+ days overdue, others still active. On paper, I looked like a serial defaulter.

Using AI to Understand the Problem#

I opened ChatGPT and uploaded the entire PDF with a simple prompt: identify everything wrong in this report.

Within minutes, it had mapped every suspicious account, flagged which ones didn’t match my history, highlighted the incorrect DOB, and explained why CIBIL systems mis-map accounts when demographic data is inconsistent.

More usefully, it drafted formal dispute letters citing relevant RBI regulations and prepared lender-specific escalations with the right legal language. It felt like having a credit compliance team on demand.

The Dispute Process#

With the AI-drafted communications as a starting point, I sent disputes to CIBIL and direct emails to each lender. The key was being specific: every email included the CIBIL report control number, the exact account identifiers from the report, and references to specific RBI regulations.

For example, when writing to Poonawalla Fincorp about a co-lending arrangement with Kissht, the email included:

CIBIL Report details:

• Control Number: [REDACTED]
• Downloaded on: [DATE]
• Where your name appears: “POONAFIN – Personal Loan – Account No. [REDACTED]”
• Delinquency trail in history: DPD values 35 / 62 / 93 / 124 during [MONTHS]

I reiterate that I have never applied for, signed, or availed any facility from Poonawalla/Kissht. This appears to be erroneous mapping / data contamination.

The emails also cited the relevant regulations explicitly:

Under Section 45-A(2) of the Credit Information Companies (Regulation) Act 2005 and Para 7.2.2 & 8.1.3 of the RBI Master Directions on Credit Information Companies (2021), please verify this record against your origination/KYC systems. If the record is not verifiable or was created with misused/incorrect KYC, immediately instruct TransUnion CIBIL to delete/correct the entry.

This kind of precise, regulation-backed language gets results. Vague complaints are likely to get ignored or deprioritized. Specific complaints with control numbers, account IDs, and regulatory citations get escalated to teams that can actually fix things.

For co-lending cases (common with fintechs like Kissht, Ring, etc.), I learned to CC both parties and explicitly request a “consolidated correction” so the entry gets fully removed rather than bouncing between two institutions.

When initial responses were slow, I sent reminders that referenced the original complaint number and the 30-day statutory deadline:

This is a reminder regarding my complaint Ref No. [REDACTED]. The acknowledgement stated that the issue would be resolved by [DATE], yet I have not received any confirmation.

Failure to resolve within the statutory period will leave me with no option but to escalate to the RBI Integrated Ombudsman.

CIBIL started closing disputes. One by one, accounts were removed. Eight fraudulent accounts were purged in the first wave.

But there was a catch: even after the fraudulent accounts were removed, my DOB was still wrong. CIBIL kept closing my DOB correction disputes without actually fixing the underlying data. Their responses were templated and generic, treating it like a lender issue when DOB is actually a CIBIL demographic field that they control directly.

This required escalating to the Nodal Officer with a sharper tone:

My Date of Birth correction dispute has been closed twice, yet my DOB remains incorrect in every new CIBIL report. This is a CIBIL demographic field — it is not lender-controlled and should have been corrected immediately once KYC was submitted.

Because of this incorrect DOB, my profile was wrongly merged with another individual’s data. Although many wrong accounts have been removed, the root cause remains uncorrected — the wrong DOB is still mapped, and therefore the risk of future wrongful linkages still exists.

Only after escalating to the Nodal Officer did the DOB finally get corrected. Once that happened, the system stopped associating the other person’s accounts with my profile. It was an algorithmic identity collision, and fixing the DOB resolved it.

The Outcome#

My latest CIBIL report shows the correct date of birth, zero fraudulent loans, no written-off or overdue accounts, and a score back in a healthy range. Only my actual accounts remain.

What I Learned#

Credit bureaus are not infallible. A single incorrect demographic detail (in my case, a mismatched DOB) can cause wrong loan mappings, score drops, false delinquencies, and a complete distortion of your financial identity.

The resolution required documentation, persistence with escalations, and an understanding of RBI regulations. AI made the last part significantly easier. Instead of spending hours researching dispute procedures and drafting formal letters, I could focus on gathering the right documents and following up with the right people.

If you haven’t checked your CIBIL report recently, it’s worth verifying that your basic details are correct: DOB, PAN, address, mobile, email. One wrong field can create problems that take weeks to untangle.

Fin!

]]> Karan Sharma https://openalgo.medium.com/how-openalgo-websocket-works-8c5e61b71d06?source=rss-cda86e929c3------2 https://openalgo.medium.com/how-openalgo-websocket-works-8c5e61b71d06?source=rss-cda86e929c3------2 Wed, 26 Nov 2025 07:42:18 GMT In active trading, speed is everything. A trade can succeed or fail depending on how quickly you receive updates about price movements, order book changes, or market depth. Many traders want the benefit of real time data without needing complicated code. OpenAlgo solves this by turning many different broker feeds into one clean and fast data stream.

OpenAlgo connects with more than twenty four Indian brokers and streams real time market data into Python strategies, Excel sheets, or the OpenAlgo web dashboard. The system lets traders focus on decision making and strategy development while the platform handles the technical work in the background.

This guide explains the complete OpenAlgo WebSocket flow from broker servers to your applications. It is written so that both traders and developers can understand the logic clearly.

Architecture Overview

One: Broker Cloud

This is where real time data begins. Each broker offers a WebSocket feed, but every broker sends data in a different style. Some use different authentication rules, some use different symbol formats, and some send different fields.

OpenAlgo supports more than twenty four brokers. They stream:

• LTP which is the last traded price
• Quote data such as open high low close volume and bid ask
• Market depth with five or twenty or fifty levels

Trader view

If you switch brokers you do not have to learn a new API or data format. The output always looks the same through OpenAlgo.

Developer view

You do not need to write separate code for each broker feed. The next layer handles that work.

Two: Broker Adapters

Broker Adapters act as translators. They take the raw broker feed and convert it into one consistent OpenAlgo format.

Tasks performed by an adapter

• Authenticate with the broker
• Connect to the broker WebSocket
• Subscribe to user selected symbols
• Convert broker specific data to the OpenAlgo format
• Publish clean data into the ZeroMQ message bus

Unified data format

You always receive a structured JSON object in one standard format.
Example:

{
  "symbol": "RELIANCE",
  "exchange": "NSE",
  "ltp": 2847.35,
  "open": 2830,
  "high": 2855,
  "low": 2825.5,
  "close": 2832.1,
  "volume": 1250000,
  "bid": 2847.3,
  "ask": 2847.4,
  "timestamp": "2024 01 15T10 30 45.123Z"
}

Trader view

No matter which broker you choose, the data is always clean and uniform.

Developer view

You write your strategy once and use it with any supported broker.

Three: ZeroMQ Message Bus

After the data is normalized it is sent to ZeroMQ which is a fast and lightweight message system.

ZeroMQ offers:

• Very low latency
• Asynchronous data flow
• Publish and subscribe messaging
• Excellent performance even with high message volumes

How the flow works

Adapters publish data into a ZeroMQ topic.
The OpenAlgo server subscribes to all topics.
ZeroMQ delivers the data quickly and efficiently.

Trader view

Charts update instantly and strategies react without delay.

Developer view

ZeroMQ removes complicated networking code and provides a simple high speed message pipeline.

Four: OpenAlgo Server

This is the center of the OpenAlgo ecosystem. It contains two main parts.

A. WebSocket Proxy Server

The WebSocket Proxy distributes real time data to all connected clients such as Python scripts, Excel sheets, or browser dashboards.

It is responsible for:

• User authentication
• Managing subscriptions for each client
• Handling multiple clients at the same time
• Listening to ZeroMQ
• Throttling updates to avoid excessive traffic

B. REST API Server

This part manages:

• Order placement
• Order updates
• Positions and holdings
• Account data
• Historical candles

Trader view

You can run the server on your laptop or on a cloud machine and keep your strategy active through the entire market session.

Developer view

You receive a clear separation between high speed streaming and business logic for orders.

Five: Client Applications

Many different types of clients can consume the OpenAlgo feed at the same time.

OpenAlgo Web Dashboard

The dashboard provides:

• Live charts
• Market watch
• Portfolio and positions
• Order window
• Real time price updates

This is ideal for traders who prefer a visual interface.

Excel Add In with OpenAlgo Functions

The Excel Add In connects Excel directly to the OpenAlgo WebSocket.
It provides custom worksheet functions created solely for OpenAlgo.
It uses a continuous WebSocket connection to keep each cell updated.

Available functions

=oa_ltp("NSE", "RELIANCE")
=oa_quotes("NSE", "NIFTY_INDEX")
=oa_depth("NFO", "BANKNIFTY30DEC25FUT")

Each function subscribes to live data and updates the cell automatically.

What traders can build

• Watchlists
• P & L trackers
• Custom option chain
• Spread sheets
• Scanners
• Risk calculators
• Alerts using Excel formulas

No programming knowledge is needed. If you know Excel you can build powerful real time tools.

Python SDK for Automated Strategies

Here is a simple Python example:

from openalgo import api

client = api(
    api_key="your_api_key",
    host="http://127.0.0.1:5000",
    ws_url="ws://127.0.0.1:8765"
)

def on_update(data):
    print(data["symbol"], data["ltp"])

client.connect()
client.subscribe_ltp(
    [{"exchange": "NSE", "symbol": "RELIANCE"}],
    on_data_received=on_update
)

This can be expanded into advanced strategies for intraday trading, positional trading, option buying, scalping, or market making.

Subscription Modes

You can choose the level of detail needed for your strategy.

LTP Mode

Fastest updates
Ideal for quick alerts and light strategies

Quote Mode

Includes OHLC volume and bid ask
Best for intraday models and chart based strategies

Market Depth Mode

Full order book
Used for scalping order flow and market making

Multiple Strategies at the Same Time

You can run several strategy scripts in parallel. Each client maintains its own WebSocket connection and subscribes to only the symbols it needs.

This provides:

• Complete isolation
• Parallel testing
• Independent risk control
• Smooth scaling

Traders often run a main live strategy while testing a new one in the background.

Performance Features

OpenAlgo includes several internal optimizations.

• Instant lookup subscription indexing
• Throttling to control message flood
• Batch broadcasting for efficiency
• Zero copy transfers inside ZeroMQ

These features ensure that your data remains fast and stable even under heavy load.

Resilience and Error Recovery

The system detects and recovers from issues such as:

• Broker feed disconnect
• Unstable internet
• Client disconnect

The server automatically reconnects and continues streaming once stable.
This keeps your strategies reliable during real market conditions.

Quick Start Python Example

from openalgo import api
import time

API_KEY = "your_api_key"
HOST = "http://127.0.0.1:5000"
WS_URL = "ws://127.0.0.1:8765"

client = api(api_key=API_KEY, host=HOST, ws_url=WS_URL)
prices = {}

def on_ltp(data):
    symbol = data["symbol"]
    prices[symbol] = data["ltp"]
    print(symbol, data["ltp"])

def main():
    instruments = [
        {"exchange": "NSE", "symbol": "RELIANCE"},
        {"exchange": "NSE", "symbol": "INFY"}
    ]

    client.connect()
    client.subscribe_ltp(instruments, on_data_received=on_ltp)

    while True:
        time.sleep(1)

if __name__ == "__main__":
    main()

Conclusion

OpenAlgo delivers consistent and ultra fast real time data to both traders and developers. It removes broker based complexity and provides one unified system for all trading workflows.

Summary for traders

• You can use Excel or the Web UI without any coding
• You get clean and accurate real time data
• You can switch brokers without changing your setup

Summary for developers

• Unified data format across all brokers
• ZeroMQ powered low latency pipeline
• Clean Python SDK for strategy development
• Support for multiple strategies at the same time

OpenAlgo gives you a solid foundation for building reliable trading setups whether you use Excel or Python or a fully automated system.

]]> Rajandran R (Creator - OpenAlgo) zmq websocket algorithmic-trading openalgo python https://openalgo.medium.com/white-box-vs-black-box-algorithms-6eacc52cc2fc?source=rss-cda86e929c3------2 https://openalgo.medium.com/white-box-vs-black-box-algorithms-6eacc52cc2fc?source=rss-cda86e929c3------2 Wed, 26 Nov 2025 03:51:48 GMT A Simple Guide for Traders

Algorithmic trading has entered the mainstream as SEBI opens the door for safer retail participation. Along with this shift, two terms have become central in the discussion. White Box algorithms and Black Box algorithms. Knowing the difference helps traders choose tools that match their comfort, transparency needs and regulatory requirements.

What is a White Box Algorithm

A White Box algorithm is transparent. The trader can see every rule and understand the logic line by line. The behaviour of the strategy is predictable because the conditions for entry and exit are visible. For example, a trader can view rules such as buy when the RSI crosses a particular threshold or exit when the price dips below a moving average.

The clarity makes these systems easier to audit. Exchanges approve them with simpler checks, and brokers can offer them directly to retail traders. White Box systems suit traders who want complete visibility of how decisions are generated.

What is a Black Box Algorithm

A Black Box algorithm keeps its internal logic hidden. Only the provider knows the exact conditions that trigger buy or sell signals. These systems often rely on proprietary models or machine learning based methods that the provider does not disclose.

Since the underlying logic is not visible, the regulatory framework is tighter. Providers must be registered as Research Analysts and maintain strategy documentation that is submitted to the exchanges. Retail traders using these systems rely on the provider’s reputation and on the approval filters put in place by SEBI.

Why This Distinction Matters

White Box systems offer simplicity and clarity. The trader can predict behaviour and understand how the algorithm responds to market signals. This also helps brokers and exchanges monitor unusual activity and maintain system integrity.

Black Box systems offer complexity and sometimes speed or statistical advantage. But they demand trust because the user cannot verify the strategy logic. This makes oversight essential and places more responsibility on the provider.

How SEBI Regulates Them

SEBI has introduced a uniform structure for all algorithms that brokers or third party platforms offer to retail traders. Every strategy must be approved by the exchange and assigned a unique identification code.
White Box systems go through a straightforward approval process because the logic is visible.

Black Box systems must pass detailed scrutiny. Any modification in logic must be reported and reapproved. The goal is to bring transparency where possible and accountability everywhere.

Where OpenAlgo Fits in the Landscape

It is important to note that OpenAlgo itself is not a trading strategy. It is an open source platform that gives traders the freedom to build their own strategies. Nothing is prebuilt or packaged inside it.

Users design their own logic and run it on their own infrastructure or connect it with tools they already use such as Amibroker, Tradingview, Metatrader, Excel, N8N and similar platforms.

Because OpenAlgo only provides the framework and not the strategy logic, the transparency or opacity of the final algorithm depends entirely on what the user creates. The platform simply enables traders to host and execute their own ideas in a flexible and self controlled environment.

Which One Should You Choose

A White Box approach is suitable for traders who want control and complete visibility.

A Black Box approach suits traders who prefer sophistication and are comfortable relying on a provider.

Open source platforms like OpenAlgo sit in the middle. They allow users to build anything they want and decide whether their logic remains visible or private. The trader chooses the level of transparency and the level of complexity.

Both paths can be effective. The choice depends on your comfort with disclosure, your faith in the provider or your wish to build and run your own strategies independently.

]]> Rajandran R (Creator - OpenAlgo) market-regulation sebi black-box white-box trading-algorithms https://openalgo.medium.com/claude-opus-4-5-nano-banana-the-ai-stack-that-turned-my-github-repo-into-a-pitch-deck-cd39f294dead?source=rss-cda86e929c3------2 https://openalgo.medium.com/claude-opus-4-5-nano-banana-the-ai-stack-that-turned-my-github-repo-into-a-pitch-deck-cd39f294dead?source=rss-cda86e929c3------2 Tue, 25 Nov 2025 06:10:20 GMT I stared at my screen, dreading the next few hours.

I needed a pitch deck for OpenAlgo — our open-source algorithmic trading platform. The kind with sleek diagrams, consistent branding, and those infographics that make traders get to know better about openalgo

The usual options flashed through my mind: PowerPoint templates. Canva. Hiring a designer. Hours of dragging boxes and aligning text.

Then I thought — what if I didn’t have to do any of that?

What if AI could read my source code and create the entire presentation for me?

Spoiler: It worked. And it worked better than I expected.

The Unexpected Combination

Two things came together this week that made this possible.

Claude Opus 4.5 — Anthropic’s most powerful model — released yesterday. Its ability to understand and reason about code is remarkable.

Fal_Marketplace — a Claude Code plugin we built last week — brings fal.ai’s image generation models directly into your coding workflow.

The combination is surprisingly powerful.

The Workflow

Here’s what the entire process looks like:

The four-stage pipeline: Source Code → Claude Opus 4.5 → Nano Banana Pro → Professional PPT

Let me break down each stage.

Stage 1: Point Claude at Your Codebase

I opened Claude Code in my OpenAlgo repository and typed:

> “Create a 30-slide PPT on Introduction to OpenAlgo. This is for traders and developers. Use the nano-banana plugin to generate process diagrams and infographics. Make it a modern startup pitch deck.”

That’s it. One prompt.

Claude didn’t ask me what OpenAlgo does. It didn’t need a feature list or marketing copy.

It read the source code.

Within seconds, it was exploring routes, API endpoints, broker integrations, and documentation files. It understood that OpenAlgo connects to 24+ Indian brokers, supports 12 trading platforms, and has features like Sandbox Testing and Action Center.

All from the code.

Stage 2: The Secret Weapon — 2000+ Word Prompts

Here’s what most people don’t realize about image generation.

Nano Banana Pro accepts prompts with over 2000 words.

This changes everything.

Basic prompts produce generic results. Detailed prompts produce designer-quality output

Instead of writing “create a flowchart,” Claude wrote prompts like this:

Create a professional architecture diagram titled “ONE API, 24+ BROKERS”

in bold white text at the top.

TOP SECTION — “YOUR TRADING PLATFORM”:

Three boxes in a row labeled: “TradingView”, “Amibroker”, “Python/Excel”

All three have arrows pointing down to a single point.

MIDDLE SECTION — “OPENALGO API LAYER”:

A large glowing rectangular box in the center labeled “OpenAlgo Unified API”

with subtitle “Single Integration Point”

Inside this box, show text: “Place Order | Modify | Cancel | Positions | Holdings | Quotes”

BOTTOM SECTION — “BROKER CONNECTIONS”:

From the API box, 24 small lines fan out downward connecting to broker names:

Row 1: “Zerodha”, “Angel One”, “Upstox”, “Dhan”, “Fyers”, “5Paisa”, “IIFL”, “Kotak”

Row 2: “Motilal”, “Groww”, “Shoonya”, “Flattrade”, “Alice Blue”, “Firstock”…

Style: Modern tech diagram, dark gradient background, neon blue and purple

accent colors, clean connecting lines, professional typography.

The result? A polished, branded diagram that looks like a designer spent hours on it.

Claude wrote ten of these detailed prompts — one for each infographic — because it understood exactly what needed to be visualized.

— -

Stage 3: Automatic Assembly

Claude didn’t stop at generating images.

It wrote a Python script using `python-pptx` that:

- Created 31 slides with 16:9 aspect ratio

- Applied a dark gradient theme with purple, cyan, and coral accents

- Set consistent typography across all slides

- Embedded each AI-generated image in the right position

- Added titles, bullet points, and captions

One command assembled everything into a polished `.pptx` file.

— -

What I Got

30 slides. Professional startup pitch deck styling.

10 custom infographics:

- Target audience personas

- Broker architecture diagram

- Platform integration ecosystem

- TradingView webhook flow

- Complete order flow

- Action Center workflow

- Sandbox testing explainer

- Strategy manager features

- Analytics dashboard

- Security features

Zero manual design work.

Total time: Under 30 minutes.

You can view the complete presentation here:

Introduction to OpenAlgo - Self Hosted Algo Trading Platform (Open Source)

Why This Matters

I’ve been thinking about what this means.

For developers: You can now create investor decks, product demos, and documentation directly from your codebase. No context-switching. No waiting for designers.

For founders: Your pitch deck actually reflects your product — because the AI read your source code, not your marketing fluff.

For educators: Any codebase can become teaching material with accurate, visual explanations.

We’re not replacing designers. We’re giving technical people the ability to create professional materials without leaving their natural environment — the code.

— -

Try It Yourself

Step 1: Install the plugin

/plugin marketplace add https://github.com/marketcalls/fal_marketplace
/plugin install nano-banana-pro

Step 2: Set your fal.ai API key

setx FAL_KEY "your-api-key-here"

Step 3: Restart Claude Code and ask

> “Create a presentation about [your project] using the nano-banana plugin for diagrams. Write detailed prompts — Nano Banana accepts 2000+ words.”

That last part is important. Tell Claude about the detailed prompt capability. The more context in the prompt, the better the output.

— -

The Uncomfortable Truth

I spent years learning PowerPoint tricks. Aligning shapes. Choosing color palettes. Formatting bullet points.

An AI just did it better in 30 minutes.

But here’s the thing — I’m not upset. I’m excited.

Because now I can focus on what actually matters: building the product, writing the code, solving the problems.

The presentation? That’s handled.

— -

Resources

Fal_Marketplace Plugin: github.com/marketcalls/fal_marketplace

Final Presentation: [SlideShare — Introduction to OpenAlgo]

OpenAlgo: https://github.com/marketcalls/openalgo

— -

If you try this workflow, I’d love to see what you create. Drop a comment or tag me.

And if you’re an algo trader looking to automate your strategies across Indian brokers — check out OpenAlgo. It’s open source and free forever.

]]> Rajandran R (Creator - OpenAlgo) pitch-deck nanobanana-ai openalgo claude-code powerpoint https://programmerlife1.wordpress.com/2025/11/16/hello-for-fediverse/ https://programmerlife1.wordpress.com/2025/11/16/hello-for-fediverse/ Sun, 16 Nov 2025 13:22:27 GMT I started exploring fediverse today

]]> Hariharan Uncategorized #fediverse newbie https://kcsrk.info/ocaml/2025/11/10/hacking/ https://kcsrk.info/ocaml/2025/11/10/hacking/ Mon, 10 Nov 2025 10:35:00 GMT How do you acquire the fundamental computer skills to hack on a complex systems project like OCaml? What’s missing and how do you go about bridging the gap?

There are many fundamental systems skills that go into working on a language like OCaml that only come with soaking in systems programming. By systems programming, I mean the ability to use tools like the command-line, editors, version control, build systems, compilers, debuggers, bash scripting, and so on. This is often something that one takes for granted when working on such projects, but is often inscrutable for new contributors, who may not have had the opportunity to develop these skills.

I struggle with this in my own research group. Students approach me to work on the OCaml compiler because they have studied OS, Compilers and Computer Architecture in class. But once they understand that working on OCaml involves actually hacking on systems, they are often lost. How do you build the compiler from source? How do you manage your changes? Do I have to build the entire compiler if I make a small change in the runtime system? The compiler crashes with a segfault – how do I debug it? Worse, the students do not even know what questions to ask, and come back with “This is all new to me, I don’t know where to begin. ChatGPT doesn’t help.”

The CS education in India often lacks a focus on these practical systems skills, which can make it challenging for new contributors to get involved in systems programming. Looking at my own past, my undergraduate CS education, like many others in India (and potentially elsewhere), had mandatory OS and Compiler Construction courses. But neither had a dedicated lab component. It is natural that these theoretical courses do not prepare the students for the practical aspects of systems programming.

I was privileged to have a computer at my school, an IBM PC AT Model 5170 and later an IBM PC 340, and surprisingly, had an education where I got to do programming from a very young age. There was lots of BASIC programming but also just tinkering with the system, learning how to use DOS, and later Windows 3.1, 95, and of course playing games (Doom and Prince of Persia, mostly). This early exposure to computers and systems programming gave me a head start. Many students, especially those from less privileged backgrounds, do not have this early exposure. They may have learned some programming, but not had the time to tinker with systems for extended periods of time.

This challenge of bridging the gap between theoretical CS education and practical systems programming skills is a common one faced by professors working in the broad systems area. The problem is compounded by the fact that these skills are difficult to teach in a traditional classroom setting—they require hands-on experience, experimentation, and often many hours of frustration and debugging. These are skills that come from doing, not from reading or watching lectures. I would be curious to hear from others about their experiences and how they have addressed this challenge.

That said, there are resources available online that can help new contributors acquire these skills. This list is biased to the areas of the compiler that I work on. I mainly work on the backend and the runtime system. The only reason I usually touch the frontend is to lower the features that I care about to the backend. Here are some I have found useful for working on the OCaml compiler:

• Systems programming
  • Course: MIT Missing Semester: This is a fantastic resource that covers a wide range of topics related to systems programming, including command-line tools, version control, editors, and more. The course is available online for free and includes video lectures, notes, and exercises. I encourage you to read the motivation for this course.
  • Course: Stanford CS45: CS45 is an extended version of the MIT course, and delves into the topics in more detail.
  • Video: CppCon 2015: Greg Law “Give me 15 minutes & I’ll change your view of GDB”: The talk explores GDB’s less-known features and sheds light on some advanced debugging techniques.
  • Tool: rr - Lightweight Recording and Deterministic Debugging: rr is a powerful tool for recording and replaying program execution, which can be invaluable for debugging complex issues in systems programming. I’ve stopped using gdb directly for anything non-trivial and have switched to rr.
• OCaml
  • Course: CS3100 Paradigms of Programming: The course covers a significant chunk of the OCaml language. You should be able to self-study the course to get a good understanding of the language. That said, the course deliberately does not cover the build system (dune), package manager (opam), command-line tools for the compiler (ocamlc, ocamlopt), editor integration (merlin, ocaml-lsp, ocamlformat), etc.
  • Book: Real World OCaml: The book has a section on the compiler and the runtime system, which gives a great overview of the memory representation, garbage collection, and other aspects of the runtime system.
• Diving deeper
  • Book: Systems Performance: Enterprise and the Cloud, 2nd Edition: This book provides an in-depth look at systems performance, covering topics such as CPU architecture, memory hierarchy, storage systems, and networking. It is a valuable resource for understanding the underlying principles of systems programming and performance optimization.
  • Book: The Garbage Collection Handbook: This book offers a comprehensive overview of garbage collection techniques, algorithms, and implementations. It is an essential resource for understanding memory management in programming languages like OCaml.
  • Book: The Art of Multiprocessor Programming: This book provides a deep dive into concurrent programming and synchronization techniques, which are crucial for understanding multi-threaded runtime systems like OCaml 5’s multicore runtime and the programming model.

I will probably keep editing this post as I find more resources. If you have suggestions for other useful resources or experiences to share, please feel free to reach out to me.

]]> KC Sivaramakrishnan https://ravidwivedi.in/posts/a-bad-day-in-malaysia/ https://ravidwivedi.in/posts/a-bad-day-in-malaysia/ Fri, 07 Nov 2025 07:25:20 GMT Continuing from where Badri and I left off in the last post. On the 7th of December 2024, we boarded a bus from Singapore to the border town of Johor Bahru in Malaysia. The bus stopped at the Singapore emigration for us to get off for the formalities.

The process was similar to the immigration at the Singapore airport. It was automatic, and we just had to scan our passports for the gates to open. Here also, we didn’t get Singapore stamps on our passports.

After we were done with the emigration, we had to find our bus. We remembered the name of the bus company and the number plate, which helped us recognize our bus. It wasn’t there already after we came out of the emigration, but it arrived soon enough, and we boarded it promptly.

From the Singapore emigration, the bus travelled a few kilometers and dropped us at Johor Bahru Sentral (JB Sentral) bus station, where we had to go through Malaysian immigration. The process was manual, unlike Singapore, and there was an immigration officer at the counter who stamped our passports (which I like) and recorded our fingerprints.

At the bus terminal, we exchanged rupees at an exchange shop to get Malaysian ringgits. We could not find any free drinking water sources on the bus terminal, so we had to buy water.

Badri later told me that Johor Bahru has a lot of data centers, which need a lot of water for cooling. When he read about it later, he immediately connected it with the fact that there was no free drinking water, and we had to buy water. Such data centers can lead to scarcity of water for others in the area.

From JB Sentral, we took a bus to Larkin Terminal, as our hotel was nearby. It was 1.5 ringgits per person (30 rupees). In order to pay for the fare, we had to put cash in a box near the driver’s seat.

Around half-an-hour later, we reached our hotel. The time was 23:30 hours. The hotel room was hot as it didn’t have air-conditioning. The weather in Malaysia is on the hotter side throughout the year. It was a budget hotel, and we paid 70 ringgits for our room.

Badri slept soon after we checked-in. I went out during the midnight at around 00:30. I was hungry, so I entered a small scale restaurant nearby, which was quite lively for the midnight hours. At the restaurant, I ordered a coffee and an omelet. I also asked for drinking water. The unique thing about that was that they put ice in hot water to make its temperature normal.

My bill from the restaurant looked like the below-mentioned table, as the items’ names were in the local language Malay:

Item	Price (Malaysian ringgits)	Conversion to Indian rupees	Comments
Nescafe Tarik	2.50	50	Coffee
Ais Kosong	0.50	10	Water
Telur Dadar	2.00	40	Omelet
SST Tax (6%)	0.30	6
Total	5.30	106

After checking out from the restaurant, I explored nearby shops. I also bought some water before going back to the hotel room.

The next day, we had a (pre-booked) bus to Kuala Lumpur. We checked out from the hotel 10 minutes after the check-out time (which was 14:00 hours). However, within those 10 minutes, the hotel staff already came up three times asking us to clear out (which we were doing as fast as possible). And finally on the third time they said our deposit was forfeit, even though it was supposed to be only for keys and towels.

The above-mentioned bus for Kuala Lumpur was from the nearby Larkin Bus Terminal. The bus terminal was right next to our hotel, so we walked till there.

Upon reaching there, we found out that the process of boarding a bus in Malaysia resembled with taking a flight. We needed to go to a counter to get our boarding passes, followed by reporting at our gate half-an-hour before the scheduled time. Furthermore, they had a separate waiting room and boarding gates. Also, there was a terminal listing buses with their arrival and departure signs. Finally, to top it off, the buses had seatbelts.

We got our boarding pass for 2 ringgits (40 rupees). After that, we proceeded to get something to eat as we were hungry. We went to a McDonald’s, but couldn’t order anything because of the long queue. We didn’t have a lot of time, so we proceeded towards our boarding gate without having anything.

The boarding gate was in a separate room, which had a vending machine. I tried to order something using my card, but the machine wasn’t working. In Malaysia, there is a custom of queueing up to board buses even before the bus has arrived. We saw it in Johor Bahru as well. The culture is so strong that they even did it in Singapore while waiting for the Johor Bahru bus!

Our bus departed at 15:30 as scheduled. The journey was around 5 hours. A couple of hours later, our bus stopped for a break. We got off the bus and went to the toilet. As we were starving (we didn’t have anything the whole day), we thought it was a good opportunity to get some snack. There was a stall selling some food. However, I had to determine which options were vegetarian. We finally settled on a cylindrical box of potato chips, labelled Mister Potato. They were 7 ringgits.

We didn’t know how long the bus is going to stop. Furthermore, eating inside buses in Malaysia is forbidden. When we went to get some coffee from the stall, our bus driver was standing there and made a face. We got an impression that he doesn’t want us to have coffee.

However, after we got into the bus, we had to wait for a long time for it to resume its journey as the driver was taking his sweet time to drink his coffee.

During the bus journey, we saw a lot of palm trees on the way. The landscape was beautiful, with good road infrastructure throughout the journey. Badri also helped me improve my blog post on obtaining Luxembourg visa in the bus.

The bus dropped us at the Terminal Bersepadu Selatan (TBS in short) in Kuala Lumpur at 21:30 hours.

Finally, we got something at the TBS. We also noticed that the TBS bus station had lockers. This gave us the idea of putting some of our luggage in the lockers later while we will be in Brunei. We had booked a cheap Air Asia ticket which doesn’t allow check-in luggage. Further, keeping the checked-in luggage in lockers for three days was cheaper than paying the excess luggage penalty for Air Asia.

We followed it up by taking a metro as our hotel was closer to a metro station. This was a bad day due to our deposit being forfeited unfairly, and got nothing to eat.

We took the metro to reach our hostel, which was located in the Bukit Bintang area. The name of this hostel was Manor by Mingle. I had stayed here earlier in February 2024 for two nights. Back then, I paid 1000 rupees per day for a dormitory bed. However, this time the same hostel was much cheaper. We got a private room for 800 rupees per day, with breakfast included. Earlier it might have been pricier due to my stay falling on weekends or maybe February has more tourists in Kuala Lumpur.

That’s it for this post. Stay tuned for our adventures in Malaysia!

]]> Ravi Dwivedi https://openalgo.medium.com/self-hosting-openais-gpt-oss-a-complete-guide-for-traders-1e85ec2a46ad?source=rss-cda86e929c3------2 https://openalgo.medium.com/self-hosting-openais-gpt-oss-a-complete-guide-for-traders-1e85ec2a46ad?source=rss-cda86e929c3------2 Wed, 05 Nov 2025 07:32:42 GMT Introduction

In August 2025, OpenAI made a significant shift by releasing GPT-OSS-20B and GPT-OSS-120B, their first open-weight language models since GPT-2. For traders and developers in the algorithmic trading space, this presents a unique opportunity to run advanced AI models privately, without API rate limits or recurring per-token costs.

This guide walks you through the complete process of setting up your own self-hosted AI infrastructure using OpenAI’s GPT-OSS-20B model. Whether you’re building trading bots, analyzing market data, or creating educational content, having a private AI instance gives you complete control over your data and infrastructure.

Why Self-Host an AI Model?

Before diving into the technical setup, let’s understand why self-hosting makes sense for trading applications:

Data Privacy: Your trading strategies, market analysis, and proprietary algorithms never leave your infrastructure. This is crucial when working with sensitive financial data.

No Rate Limits: Unlike API-based services that throttle requests or charge per token, your self-hosted model runs as much as you need without restrictions.

Cost Predictability: You pay a fixed hourly rate for the GPU server regardless of usage. For high-volume applications processing hundreds of requests daily, this becomes significantly more economical than pay-per-token APIs.

Customization: Open-weight models can be fine-tuned on your specific trading data, terminology, and use cases, providing better results than generic models.

Zero Latency: Running the model on your own infrastructure eliminates internet round-trip time, crucial for time-sensitive trading applications.

Understanding the Architecture

Our setup consists of four main components:

vLLM: A high-performance inference engine optimized for large language models. It handles GPU memory management, batching, and serves an OpenAI-compatible API.

GPT-OSS-20B: OpenAI’s open-weight model with 21 billion parameters. It uses a mixture-of-experts architecture with 3.6 billion active parameters, providing strong reasoning capabilities while remaining efficient.

Open-WebUI: A modern, ChatGPT-like web interface that connects to your vLLM server, providing an intuitive chat experience.

Docker: Containerization platform that simplifies deploying Open-WebUI with all its dependencies.

Prerequisites and Costs

Hardware Requirements

For GPT-OSS-20B, you need:

• GPU with at least 16GB VRAM (the model uses approximately 14GB)
• 8+ CPU cores recommended
• 32GB+ system RAM
• 100GB+ storage for model weights and system

For this guide, we use an NVIDIA H200 GPU with 141GB VRAM, which provides headroom for the larger GPT-OSS-120B model if needed later.

Estimated Costs

Using Datacrunch cloud infrastructure:

• H200 GPU instance: $2.59 per hour
• Storage (300GB): $0.082 per hour
• Total: approximately $2.67 per hour

If running 24/7, this equals roughly $1,951 per month. However, you can stop the instance when not in use, as billing is per minute. Running 8 hours daily costs approximately $642 per month.

Software Requirements

All software components are open-source and free:

• Ubuntu 24.04 (included with GPU instance)
• CUDA 12.8 (pre-installed)
• Docker (free)
• vLLM (open-source)
• GPT-OSS-20B model weights (Apache 2.0 license)
• Open-WebUI (open-source)

Step 1: Provisioning Your GPU Server

We use Datacrunch for GPU infrastructure due to competitive pricing and straightforward deployment.

Create an account at https://cloud.datacrunch.io and provision a new instance with these specifications:

• Instance type: H200 (or H100 for a more budget-friendly option)
• Operating system: Ubuntu 24.04 with CUDA 12.8
• Storage: 300GB
• Location: Choose the closest data center for lowest latency

Once provisioned, you will receive:

• IP address (we’ll use xx.xx.xx.xx as placeholder)

Note : you need to create SSH key and apply the key while creating the GPU Cloud server. You’ll use it to connect to your server.

Step 2: Initial Server Setup

Connect to your server using SSH:

ssh -i gpukey root@xx.xx.xx.xx

First, verify your GPU is detected correctly:

nvidia-smi

You should see your H200 GPU listed with approximately 141GB of memory.

Update system packages:

apt update && apt upgrade -y

Step 3: Firewall Configuration

Security is crucial when running public-facing AI infrastructure. Configure UFW (Uncomplicated Firewall) to allow only necessary ports:

# Install UFW if not present
apt install ufw -y

# Default policies
ufw default deny incoming
ufw default allow outgoing

# Allow SSH (critical - do this first!)
ufw allow 22/tcp

# Allow vLLM API
ufw allow 8000/tcp

# Allow Open-WebUI
ufw allow 3000/tcp

# Enable firewall
ufw enable

# Verify configuration
ufw status

Your output should show:

Status: active

(venv) root@loud-tree-begins-ice-01:~/gpt-20b-server# ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere                   
22                         ALLOW       Anywhere
8000                       ALLOW       Anywhere
3000                       ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)
22 (v6)                    ALLOW       Anywhere (v6)
8000 (v6)                  ALLOW       Anywhere (v6)
3000 (v6)                  ALLOW       Anywhere (v6)

(venv) root@loud-tree-begins-ice-01:~/gpt-20b-server#

Step 4: Installing Docker

Open-WebUI runs in a Docker container for simplified deployment.

Install Docker:

# Add Docker's official GPG key
apt-get update
apt-get install ca-certificates curl -y
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

# Add repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null
# Install Docker
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Verify Docker installation:

docker --version

(venv) root@loud-tree-begins-ice-01:~/gpt-20b-server# docker --version
Docker version 28.5.1, build e180ab8

Verify Cuda Version

nvcc --version

(venv) root@loud-tree-begins-ice-01:~/gpt-20b-server# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0
(venv) root@loud-tree-begins-ice-01:~/gpt-20b-server# 

Install NVIDIA Container Toolkit to give Docker containers GPU access:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt-get update
apt-get install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Test GPU access from Docker:

docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

If you see your GPU information, Docker is correctly configured.

Step 5: Setting Up Python Environment for vLLM

Create a dedicated directory and Python virtual environment:

mkdir -p ~/gpt-20b-server
cd ~/gpt-20b-server

Install Python virtual environment tools:

apt install python3-venv python3-pip -y

Create and activate virtual environment:

python3 -m venv venv
source venv/bin/activate

Your prompt should now show (venv) indicating the virtual environment is active.

Install vLLM and dependencies:

pip install --upgrade pip
pip install vllm
pip install openai-harmony

The installation takes several minutes as it downloads CUDA libraries and PyTorch.

Step 6: Downloading GPT-OSS-20B Model

The model weights are hosted on Hugging Face and total approximately 41GB.

Install Hugging Face CLI:

pip install huggingface-hub

Create a directory for model storage:

mkdir -p ~/models

Download GPT-OSS-20B:

huggingface-cli download openai/gpt-oss-20b --local-dir ~/models/gpt-oss-20b

This download takes 10–15 minutes depending on your connection speed. The CLI shows progress as it downloads 18 files including model weights, tokenizer, and configuration files.

Step 7: Launching vLLM Server

With the model downloaded, start the vLLM inference server.

Set required environment variable:

# Export environment variable
export VLLM_USE_FLASHINFER_SAMPLER=0

Start vLLM in background:

# Start vLLM server
nohup python -m vllm.entrypoints.openai.api_server \
  --model openai/gpt-oss-20b \
  --download-dir ~/models \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --gpu-memory-utilization 0.9 \
  --max-model-len 32768 \
  --trust-remote-code \
  > vllm-server.log 2>&1 &

PID=$!
echo "📋 Process ID: $PID"
echo "⏳ Server is starting (takes 2-3 minutes)..."
echo ""

Configuration parameters explained:

• --model: Specifies which model to load
• --download-dir: Where model files are stored
• --host 0.0.0.0: Listen on all network interfaces
• --port 8000: API port
• --dtype auto: Automatically select optimal precision
• --gpu-memory-utilization 0.5: Use 50% of GPU memory (conservative for 20B model)
• --max-model-len 8192: Maximum context window size
• --trust-remote-code: Required for GPT-OSS custom code

Monitor the startup process:

tail -f vllm-gptoss.log

Wait for the message “Application startup complete.” This takes 2–3 minutes as the model loads into GPU memory.

Press Ctrl+C to exit the log viewer. The server continues running in the background.

Step 8: Verifying vLLM API

Test that the API is responding correctly:

curl http://localhost:8000/health

Expected output: {"status":"ok"}

Check available models:

curl http://localhost:8000/v1/models

You should see JSON output listing openai/gpt-oss-20b as an available model.

Test a simple completion:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [
      {"role": "user", "content": "What is algorithmic trading?"}
    ],
    "max_tokens": 200
  }'

If you receive a JSON response with generated text, your vLLM server is working correctly.

Step 9: Deploying Open-WebUI

Open-WebUI provides a modern chat interface similar to ChatGPT.

Start the Open-WebUI Docker container:

docker run -d \
  --name open-webui \
  --gpus all \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
  -e OPENAI_API_KEY=dummy \
  -e WEBUI_AUTH=False \
  ghcr.io/open-webui/open-webui:cuda

Configuration breakdown:

• --name open-webui: Container name
• --gpus all: GPU access (for potential future features)
• -p 3000:8080: Map container port 8080 to host port 3000
• --add-host: Allows container to reach host's localhost
• -v open-webui: Persistent storage for conversations
• -e OPENAI_API_BASE_URL: Point to your vLLM server
• -e OPENAI_API_KEY: Dummy key (vLLM doesn't require authentication)
• -e WEBUI_AUTH=False: Disable login for simplicity

Wait 15–20 seconds for the container to start, then check status:

docker ps

You should see the open-webui container with status "healthy".

Step 10: Accessing Your AI Interface

Open your web browser and navigate to:

http://xx.xx.xx.xx:3000

Replace xx.xx.xx.xx with your actual server IP address.

You should see the Open-WebUI interface. Click on the model selector dropdown at the top of the page. You should see “openai/gpt-oss-20b” listed.

Select the model and start chatting. Try a test prompt:

Explain the difference between momentum and mean reversion trading strategies.

The first response takes 15–20 seconds as CUDA performs initial warmup. Subsequent responses are much faster, typically 5–10 seconds.

Understanding Performance Metrics

vLLM automatically logs performance metrics. View them in real-time:

tail -f ~/gpt-20b-server/vllm-gptoss.log | grep "throughput"

Key metrics to watch:

Prompt Throughput: Speed of processing your input text, typically 200–300 tokens per second.

Generation Throughput: Speed of generating output text. For GPT-OSS-20B on H200, expect 100–120 tokens per second.

GPU KV Cache Usage: Memory used for caching previous tokens in conversation. Should stay below 10% for optimal performance.

These metrics help you understand system performance and identify bottlenecks.

Configuring VS Code Integration

For developers who want AI assistance while coding, integrate with VS Code using the Continue extension.

Install the Continue extension from VS Code marketplace.

Open Continue settings and edit config.json:

{
  "models": [
    {
      "title": "GPT-OSS-20B",
      "provider": "openai",
      "model": "openai/gpt-oss-20b",
      "apiKey": "dummy",
      "apiBase": "http://xx.xx.xx.xx:8000/v1",
      "contextLength": 32768,
      "completionOptions": {
        "maxTokens": 8000,
        "temperature": 0.7
      }
    }
  ]
}

Replace xx.xx.xx.xx with your server IP. Save the configuration and restart VS Code.

You can now use your self-hosted model directly within your code editor.

Use Cases for Trading Applications

With your AI infrastructure running, here are practical applications for algorithmic trading:

Strategy Documentation: Generate comprehensive documentation for your trading strategies, explaining logic, parameters, and risk management in plain English.

Code Generation: Create boilerplate code for backtesting frameworks, data pipelines, or API integrations. For example: “Write a Python function to calculate Bollinger Bands for a given price series.”

Market Analysis: Analyze earnings reports, news sentiment, or market commentary. Feed the model text data and ask for structured insights.

Educational Content: Generate explanations of complex trading concepts for courses or blog posts. The model understands financial terminology and can explain at various expertise levels.

Data Transformation: Convert between different data formats, clean datasets, or generate test data for backtesting systems.

Debugging Assistant: Paste error messages or problematic code and ask for debugging suggestions, particularly useful for complex algorithmic strategies.

API Documentation: Generate API documentation for your trading systems or explain third-party API endpoints in simple terms.

Reasoning Levels in GPT-OSS

GPT-OSS supports three reasoning levels that can be specified in system prompts:

Low Reasoning: Fast responses for simple questions. Use when speed is more important than depth. Example: “Reasoning: low”

Medium Reasoning: Balanced approach with good quality and acceptable speed. This is the default. Example: “Reasoning: medium”

High Reasoning: Deep analysis for complex problems. Takes longer but provides more thorough answers. Example: “Reasoning: high”

Set the reasoning level in your system prompt:

{
  "role": "system",
  "content": "Reasoning: high\n\nYou are a trading strategy analyst."
}

Monitoring and Maintenance

Check vLLM Status

ps aux | grep vllm

If the process isn’t running, restart it using the command from Step 7.

View Recent Logs

tail -100 ~/gpt-20b-server/vllm-gptoss.log

Check GPU Memory Usage

nvidia-smi

Monitor GPU utilization and memory. GPT-OSS-20B should use approximately 14GB of VRAM.

Check Open-WebUI Status

docker ps -a | grep open-webui

If the container stopped, restart it:

docker restart open-webui

View Open-WebUI Logs

docker logs open-webui | tail -50

Upgrading to GPT-OSS-120B

If you need higher quality reasoning and have sufficient VRAM (80GB minimum), upgrade to the larger model.

Stop the current vLLM server:

pkill -f vllm

Download GPT-OSS-120B:

cd ~/gpt-20b-server
source venv/bin/activate
huggingface-cli download openai/gpt-oss-120b --local-dir ~/models/gpt-oss-120b

Start vLLM with the larger model:

# Export environment variable
export VLLM_USE_FLASHINFER_SAMPLER=0

# Start vLLM server
nohup python -m vllm.entrypoints.openai.api_server \
  --model openai/gpt-oss-120b \
  --download-dir ~/models \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --gpu-memory-utilization 0.9 \
  --max-model-len 32768 \
  --trust-remote-code \
  > vllm-server.log 2>&1 &

PID=$!
echo "📋 Process ID: $PID"
echo "⏳ Server is starting (takes 2-3 minutes)..."
echo ""

The 120B model provides near-GPT-4o quality at the cost of slower inference (60–80 tokens per second vs 100–120 for the 20B model).

Troubleshooting Common Issues

Issue: Cannot Connect to Web Interface

Check if Open-WebUI is running:

docker ps | grep open-webui

Verify firewall allows port 3000:

ufw status | grep 3000

Check Open-WebUI logs for errors:

docker logs open-webui

Issue: Model Not Appearing in Dropdown

Refresh your browser with Ctrl+F5 (hard refresh).

Verify vLLM is running:

curl http://localhost:8000/v1/models

Check Open-WebUI can reach vLLM:

docker exec open-webui curl http://host.docker.internal:8000/v1/models

Issue: Slow Response Times

Check GPU utilization during inference:

watch -n 1 nvidia-smi

If GPU utilization is low, increase batch size or concurrent requests. If it’s maxed out, your model is at capacity.

Review vLLM logs for performance metrics:

tail -f ~/gpt-20b-server/vllm-gptoss.log | grep "Engine"

Issue: Out of Memory Errors

Reduce --gpu-memory-utilization parameter:

--gpu-memory-utilization 0.4

Or decrease --max-model-len:

--max-model-len 4096

Issue: vLLM Won’t Start

Check logs for specific errors:

cat ~/gpt-20b-server/vllm-gptoss.log | grep -i error

Common causes include insufficient disk space, incorrect model path, or CUDA version mismatches.

Cost Optimization Strategies

Stop Instance When Not in Use

Datacrunch bills per minute. Stop your instance during non-working hours:

# On your local machine
ssh root@xx.xx.xx.xx "shutdown -h now"

Running 8 hours daily instead of 24/7 saves approximately $1,300 monthly.

Use Longer Commitments

Datacrunch offers discounts for longer contracts:

• 1 month: 2% discount
• 3 months: 3% discount
• 6 months: 4% discount
• 1 year: 8% discount
• 2 years: 25% discount

A 1-year commitment reduces costs from $1,951 to $1,737 monthly, saving $214 per month.

Consider Smaller GPUs

If GPT-OSS-20B meets your needs, an H100 (80GB) at $2.048/hour saves 21% compared to the H200, or approximately $375 monthly for 24/7 operation.

Use Spot Instances

Some providers offer spot instances at 50–70% discounts. However, these can be interrupted with short notice. Suitable for non-production workloads.

Security Considerations

Change Default Ports

For production deployments, consider non-standard ports:

ufw delete allow 8000/tcp
ufw delete allow 3000/tcp
ufw allow 18000/tcp  # Custom vLLM port
ufw allow 13000/tcp  # Custom WebUI port

Update your vLLM and Docker run commands accordingly.

Enable Authentication

Open-WebUI supports authentication. Remove the WEBUI_AUTH=False flag and configure user accounts through the web interface.

Use Reverse Proxy

For production, place Nginx in front of your services:

apt install nginx -y

Configure Nginx to handle SSL/TLS, rate limiting, and access controls.

Restrict SSH Access

Limit SSH to specific IP addresses:

ufw delete allow 22/tcp
ufw allow from YOUR_IP_ADDRESS to any port 22

Regular Updates

Keep your system updated:

apt update && apt upgrade -y

Update vLLM regularly:

cd ~/gpt-20b-server
source venv/bin/activate
pip install --upgrade vllm

Credits and Acknowledgments

This guide builds upon the work and contributions of several individuals and projects:

OpenAI: For releasing GPT-OSS-20B and GPT-OSS-120B under the permissive Apache 2.0 license, making powerful AI accessible to everyone.

1littlecoder: YouTube creator whose cutting-edge AI tutorials and practical demonstrations helped validate these approaches and educated the broader community on working with modern LLMs.

vLLM Development Team: For creating and maintaining the high-performance inference engine that makes running large models practical and efficient.

Open-WebUI Team: For developing the excellent web interface that brings a professional ChatGPT-like experience to self-hosted models.

Hugging Face: For hosting model weights and providing infrastructure that makes distributing open-source models seamless.

Datacrunch: For providing competitively priced GPU infrastructure with modern hardware and developer-friendly features.

Docker and NVIDIA: For containerization technology and CUDA toolkit that simplifies deployment and GPU access.

Additional Resources

Official Documentation

• OpenAI GPT-OSS Model Card: https://openai.com/index/gpt-oss-model-card/
• vLLM Documentation: https://docs.vllm.ai
• Open-WebUI Documentation: https://docs.openwebui.com
• Hugging Face GPT-OSS Hub: https://huggingface.co/openai/gpt-oss-20b

Community Resources

• vLLM GitHub: https://github.com/vllm-project/vllm
• Open-WebUI GitHub: https://github.com/open-webui/open-webui
• OpenAI GPT-OSS Announcement: https://openai.com/index/introducing-gpt-oss/

Learning Materials

• 1littlecoder YouTube Channel: Search for latest AI/ML tutorials
• Hugging Face Blog on GPT-OSS: https://huggingface.co/blog/welcome-openai-gpt-oss

Conclusion

Self-hosting AI infrastructure provides traders and developers with unprecedented control, privacy, and cost efficiency. With GPT-OSS-20B, you have access to a capable reasoning model that handles everything from code generation to market analysis without external dependencies.

The initial setup requires some technical investment, but the result is a production-ready system that serves unlimited requests at a fixed cost. For high-volume applications, trading strategy development, or educational content creation, this approach offers significant advantages over API-based services.

As open-source AI continues to evolve, having your own infrastructure positions you to experiment with new models, fine-tune on proprietary data, and build sophisticated AI-powered trading tools without vendor lock-in.

The complete setup outlined in this guide provides a solid foundation. From here, you can explore fine-tuning models on your trading data, building RAG systems with your research, or integrating AI into your existing trading infrastructure.

Whether you’re developing algorithmic strategies, creating educational content, or building trading tools, self-hosted AI gives you the flexibility and control to innovate without constraints.

Disclaimer

This article is for educational purposes. Trading involves substantial risk of loss. AI-generated content should not be considered financial advice. Always verify AI outputs and conduct thorough testing before deploying any trading strategies.

]]> Rajandran R (Creator - OpenAlgo) gpt-oss cline-extension self-hosting vllm open-source https://openalgo.medium.com/building-an-agentic-trader-from-scratch-a-beginners-guide-bb74b10438b4?source=rss-cda86e929c3------2 https://openalgo.medium.com/building-an-agentic-trader-from-scratch-a-beginners-guide-bb74b10438b4?source=rss-cda86e929c3------2 Mon, 03 Nov 2025 18:16:12 GMT Imagine having an AI assistant that watches the stock market 24/7, analyzes multiple technical indicators, makes trading decisions, and executes trades automatically. This isn’t science fiction — it’s what we call an “agentic trader.” In this article, I’ll walk you through how to build one from scratch, even if you’re just starting out.

What is an Agentic Trader?

An agentic trader is an AI-powered autonomous system that can make and execute trading decisions without constant human intervention. Think of it as a robot trader that:

- Monitors market data in real-time

- Analyzes technical indicators (like RSI, MACD, Bollinger Bands)

- Makes BUY/SELL/HOLD decisions based on patterns

- Executes trades automatically through a broker API

- Learns from past trades to improve over time

The key word here is “agentic” — meaning it has agency. It can take actions on its own within the rules you set.

Understanding the Building Blocks

Before we dive into building, let’s understand the three key technologies that make this possible:

1. OpenAI Agents SDK: The Brain

The OpenAI Agents SDK is a framework that lets you build AI agents — programs that can think, plan, and take actions. Here’s what makes it special:

What it does:

- Gives your AI the ability to use tools (like checking stock prices, placing orders)

- Allows the AI to plan multi-step actions (“First check the price, then if it’s below $100, buy 10 shares”)

- Handles the back-and-forth conversation between the AI and your tools

- Manages memory so the agent remembers previous trades.

Why it’s powerful:

Think of it like giving your AI hands and feet. Without the SDK, an AI model like GPT-5 can only generate text. With the SDK, it can actually DO things — check your account balance, fetch stock prices, place orders, and more.

Example:

Without SDK: AI can only suggest

AI: “I think you should buy RELIANCE stock”

With SDK: AI can actually execute

AI: *checks price* “RELIANCE is at 1385”

AI: *checks your balance* “You have sufficient funds”

AI: *places order* “Bought 10 shares of RELIANCE at 1385”

2. LiteLLM: The Universal Translator

LiteLLM is like a universal adapter for AI models. Here’s why it matters:

The Problem:

Different AI providers (OpenAI, Google, Anthropic, Groq, Cerebras) all have different APIs. If you build your agent to work with OpenAI’s GPT-5, switching to a faster or cheaper model from another provider means rewriting your code.

The Solution:

LiteLLM translates between all these different APIs into one standard format. Write your code once, and it works with 100+ different AI models.

Why this matters for trading:

Different models have different strengths:

- OpenAI GPT-5: High quality reasoning, but slower and expensive

- Cerebras GPT OSS-120B: Ultra-fast (2000+ tokens/second), great for real-time trading

- Groq Llama 3.3: Fast and cheap ($0.59 per 1M tokens), perfect for testing

- Claude 4.5 Sonnet: Excellent at complex analysis, but pricey

With LiteLLM, you can switch between them by changing just ONE line in your config file:

# .env file
MODEL_PROVIDER=cerebras # Ultra-fast for production
MODEL_PROVIDER=groq # Cheap for testing
MODEL_PROVIDER=openai # High quality reasoning

No code changes needed. Just swap and go.

3. Why Build Model-Agnostic?

Being model-agnostic (able to work with any AI model) gives you superpowers:

Cost Optimization:

- Development: Use cheap Groq models ($0.59/million tokens)

- Production: Switch to fast Cerebras models ($0.60/million tokens)

- Analysis: Use powerful Claude models when you need deep reasoning

Speed Matters:

In trading, milliseconds count. With model-agnostic design:

- OpenAI GPT-5-mini: 3–5 seconds per trading decision

- Cerebras Llama 3.1: 0.5–1 second per decision (80% faster!)

Fallback Strategy:

If one provider has downtime, instantly switch to another. Your trading doesn’t stop.

Future-Proof:

New, better models come out every month. With LiteLLM, you can test them immediately without changing your code.

Token Efficiency: Why It Matters

Here’s something most beginners don’t realize: AI models charge you by the “token” (roughly a word or piece of a word). Every message you send and every response costs money.

The Token Efficiency Lesson

When I first built my agentic trader, I made a common mistake: using a multi-agent architecture (multiple specialized AI agents working together). Here’s what I learned:

Multi-Agent Architecture:

- Market Analysis Agent (analyzes charts)

- Risk Management Agent (checks if trade is safe)

- Execution Agent (places orders)

- Memory Agent (remembers past trades)

- Coordinator Agent (manages all the others)

Sounds smart, right? But here’s the cost:

Per Trading Cycle:
- Input tokens: 144,000
- Cost: $0.023
- Time: 12–20 seconds
- Daily cost (75 cycles): $1.73
- Monthly cost: ~$52

**Single Agent Architecture:**

One smart agent that does everything:

Per Trading Cycle:
- Input tokens: 30,000 (79% less!)
- Cost: $0.008 (65% cheaper!)
- Time: 3–6 seconds (70% faster!)
- Daily cost (75 cycles): $0.60
- Monthly cost: ~$18

The Lesson:

More agents ≠ Better results. In fact, simpler is often better. Each time agents talk to each other, you pay for tokens. A well-designed single agent is faster, cheaper, and easier to debug.

Building the Agentic Trader Prototype

Now let’s talk about what our prototype actually does. This is an educational system designed to teach you how agentic trading works — NOT for live trading.

What the Prototype Does

1. Market Analysis:

The agent analyzes 5 NSE stocks (ICICIBANK, RELIANCE, SBIN, WIPRO, ITC) using 7 professional technical indicators:

- RSI (Relative Strength Index): Is the stock overbought or oversold?

- MACD (Moving Average Convergence Divergence): What’s the momentum?

- Bollinger Bands: Is the price at an extreme?

- EMA (Exponential Moving Average): What’s the trend direction?

- Stochastic Oscillator: Are we at a reversal point?

- ADX (Average Directional Index): How strong is the trend?

- ATR (Average True Range): How volatile is the stock?

The agent fetches all this data in parallel (all 5 stocks at once) in just 2–3 seconds.

2. Decision Making:

The agent looks for alignment across indicators. It only trades when 3+ indicators agree:

Example Decision Process:
ICICIBANK:
- RSI: 28 (oversold) → BUY signal
- MACD: Bullish crossover → BUY signal
- Price: Near lower Bollinger Band → BUY signal
- EMA: Price above 20 EMA → BUY signal
- Stochastic: < 20 → BUY signal
Result: 5 aligned signals → STRONG BUY
Action: Place market order for 7 shares (Rs.10,000 investment)

3. Risk Management:

The agent has built-in safety features:

- Daily Stop-Loss: Stops trading if you lose more than Rs.10,000 in a day

- Trade Limits: Maximum 5 trades per stock per day

- Position Control: Won’t add to existing positions (no pyramiding)

- Market Hours: Only trades 9:15 AM — 3:15 PM IST

- Auto Square-Off: Closes all positions at 3:15 PM

4. Memory & Learning:

The agent remembers:

- Every trade it made

- Profit/loss per trade

- Which indicators worked best

- Market conditions during each trade

This helps it improve over time (though remember, this is a learning prototype, not a money-making guarantee).

5. Account Access:

Through OpenAlgo, the agent can:

- Check your account balance

- View open positions

- See profit/loss in real-time

- Place market orders

- Cancel pending orders

- Close positions

Setting Up: What You Need

To build and run this prototype, you need:

1. OpenAlgo Running on Your System

What is OpenAlgo?

OpenAlgo is an open-source bridge between your code and your stock broker. It translates your code commands into broker-specific API calls.

Setup Steps:

# Install OpenAlgo from

git clone https://github.com/marketcalls/openalgo

OpenAlgo by default runs on: http://127.0.0.1:5000

Connect Your Broker:

- Log into OpenAlgo dashboard

- Connect your broker account (Zerodha, Fyers, Dhan, Upstox, etc.)

- Generate an API key

- Save the API key for your trading agent

Why OpenAlgo?

Without it, you’d need to write broker-specific code for each broker. OpenAlgo gives you one universal API that works with 23+ Indian brokers.

2. Get an LLM API Key

You need an API key from at least one AI model provider. Here are your options:

For Beginners (Cheapest):

- **Groq**: Free tier available, very fast

- Get key at: https://console.groq.com/

- Model: llama-3.3–70b-versatile

- Cost: $0.59 per 1M tokens

For Production (Fastest):

- Cerebras: Ultra-fast inference

- Get key at: https://cloud.cerebras.ai/

- Model: llama3.1–8b

- Cost: $0.60 per 1M tokens

- Speed: 1800+ tokens/second

For Quality (Most Accurate):

- OpenAI: Industry standard

- Get key at: https://platform.openai.com/

- Model: gpt-5-mini

- Cost: $0.15 input / $0.60 output per 1M tokens

Recommendation for Learning:

Start with Groq. It’s fast, cheap, and perfect for testing. Once you understand how everything works, you can switch to Cerebras for speed or OpenAI for quality.

3. Install the Agentic Trader

# Clone the repository

git clone https://github.com/marketcalls/Agentic-Trader.git
cd Agentic-Trader
# Install uv package manager (fast Python installer)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install dependencies
uv sync
# Configure environment
cp .env.example .env

Edit the `.env` file:

# Choose your AI provider
MODEL_PROVIDER=groq
# Add your AI API key
GROQ_API_KEY=gsk-your-key-here
GROQ_MODEL=groq/llama-3.3–70b-versatile
# Add your OpenAlgo API key
OPENALGO_API_KEY=your-openalgo-key-here
OPENALGO_HOST=http://127.0.0.1:5000

4. Run the Agent

# Start the agent
uv run python agent.py

You’ll see output like:

================================================================================
[INIT] Initializing Trading State…
================================================================================
[INIT] Fetching account funds…
[INIT] ✓ Available Cash: Rs.10,500.00
[INIT] Fetching open positions…
[INIT] ✓ Open Positions: 0
[INIT] ✓ Daily P&L: Rs.0.00
[INIT] ✓ Stop-loss check: OK
================================================================================
Trading Cycle: 2025–01–15 10:30:00 IST
================================================================================
[BULK DATA] Fetching data for 5 symbols in parallel…
[BULK DATA] Completed in 2.3 seconds
ICICIBANK: BUY Order#123 (MACD bullish)
RELIANCE: HOLD (weak signals)
SBIN: HOLD (existing position)
WIPRO: SELL Order#124 (take profit)
ITC: HOLD (mixed signals)
[TOKEN USAGE]
Total Tokens: 31,085
Cost: $0.008

How It All Works Together

Let me walk you through a complete trading cycle:

Step 1: Initialization (Happens Once at Startup)

Agent starts → Connects to OpenAlgo → Checks account balance
→ Fetches open positions → Calculates current P&L
→ Verifies stop-loss status → Ready to trade

Step 2: Market Data Fetch (Every 5 Minutes)

Agent wakes up → Fetches data for all 5 stocks in parallel
→ Gets quotes, depth, historical data
→ Calculates 7 technical indicators for each stock
→ Completes in 2–3 seconds

Step 3: Analysis (For Each Stock)

Agent analyzes ICICIBANK:
→ RSI = 28 (oversold)
→ MACD = bullish crossover
→ Price near lower Bollinger Band
→ 3+ indicators aligned → BUY signal
Agent checks risk constraints:
→ Daily P&L = Rs.250 (OK, below stop-loss)
→ Trades today = 2 (OK, below limit of 5)
→ Existing position? No (OK to trade)
→ Risk check PASSED
Agent calculates position size:
→ LTP = Rs.1,346.40
→ Investment = Rs.10,000
→ Quantity = 7 shares
→ Actual cost = Rs.9,424.80

Step 4: Execution

Agent places market order:
→ Symbol: ICICIBANK
→ Action: BUY
→ Quantity: 7
→ Order Type: MARKET (instant execution)
→ Strategy: AI Agent
OpenAlgo sends order to broker
→ Broker executes immediately at market price
→ Order ID: #123
→ Status: COMPLETED

Step 5: Tracking

Agent updates state:
→ Records trade in history
→ Updates trade count (now 3 for ICICIBANK)
→ Calculates new P&L
→ Saves to memory for learning

**Step 6: Wait**

Agent sleeps for 5 minutes → Repeat from Step 2

**Step 7: Square-Off (3:15 PM)**

Market closing time → Agent checks all open positions
→ ICICIBANK: 7 shares long → Place SELL 7 shares
→ RELIANCE: 1 share short → Place BUY 1 share
→ All positions closed
→ Cancel any pending orders
→ Done for the day

## Understanding the Architecture

Here’s what makes this prototype token-efficient:

### Single Agent Flow

┌─────────────────────────────────────┐
│ AUTONOMOUS TRADING AGENT │
│ (One Smart Agent) │
└─────────────────┬───────────────────┘
│
▼
┌─────────────────────────────────┐
│ BULK DATA FETCH (Parallel) │
│ • All 5 stocks at once │
│ • 2–3 seconds total │
└─────────────┬───────────────────┘
│
▼
┌─────────────────────────────────┐
│ ANALYZE EACH STOCK │
│ • Check 7 indicators │
│ • Make BUY/SELL/HOLD decision │
└─────────────┬───────────────────┘
│
▼
┌─────────────────────────────────┐
│ VALIDATE & EXECUTE │
│ • Check risk constraints │
│ • Calculate position size │
│ • Place orders in bulk │
└─────────────┬───────────────────┘
│
▼
┌─────────────────────────────────┐
│ OPENALGO → BROKER │
│ • Orders executed │
└─────────────────────────────────┘

Why This is Efficient

Single Context:

Instead of passing data between multiple agents (which costs tokens), everything happens in one conversation. The agent gets all the data once, makes all decisions, and executes all orders in one go.

Bulk Operations:

- Fetches data for ALL stocks in parallel (not one by one)

- Places ALL orders at once (not sequentially)

- Checks ALL risk constraints in one batch

Minimal Prompts:

The agent’s instructions are just 30 lines of plain text, not 80+ lines of verbose explanations.

No Memory Bloat:

Each trading cycle is independent. The agent doesn’t carry forward massive conversation history.

Real-World Performance

Let me show you actual numbers from testing:

Speed

Total Cycle Time: 5-9 seconds
Breakdown:
- Data Fetch (5 stocks): 2–3s
- Analysis & Decisions: 2–4s
- Order Execution: <1s

Compare this to the multi-agent version: 12–20 seconds per cycle.

### Cost

Per Trading Cycle:
- Tokens Used: ~30,000
- Cost (Groq): $0.018
- Cost (Cerebras): $0.018
- Cost (OpenAI): $0.008
Daily (75 cycles):
- Groq: $1.35
- Cerebras: $1.35
- OpenAI: $0.60
Monthly:
- Groq: ~$40
- Cerebras: ~$40
- OpenAI: ~$18

Accuracy

The agent requires 3+ indicators to align before trading. In testing:

- Win Rate: ~55–60% (varies by market conditions)

- Risk-Adjusted: All trades limited to Rs.10,000

- Max Loss: Rs.10,000 per day (enforced by code)

Important: These are prototype testing results, not live trading performance. Past performance doesn’t guarantee future results.

What You Should Know

This is an Educational Prototype

Let me be very clear about what this is and isn’t:

What it IS:

- A learning tool to understand agentic AI

- A demonstration of token-efficient architecture

- A way to experiment with AI trading concepts

- Open source code you can study and modify

What it is NOT:

- A guarantee to make money

- A production-ready trading system

- Financial advice

- A get-rich-quick scheme

Important Safety Points

1. Paper Trade First:

Test with fake money before real money. OpenAlgo supports paper trading mode.

2. Start Small:

If you do test with real money, start with the minimum amount you can afford to lose completely.

3. Understand the Risks:

- Markets are unpredictable

- Technical indicators can give false signals

- AI can make mistakes

- Past performance ≠ future results

- You can lose all your capital

4. Legal Compliance:

Check if algorithmic trading is legal in your jurisdiction. In India, it’s allowed for retail traders, but always verify.

5. Monitor It:

Never leave an automated trading system running unattended for long periods. Check it regularly.

Learning Path: Where to Go From Here

If you want to build on this prototype, here’s a learning path:

Beginner Level

- Run the prototype in paper trading mode

- Understand how each function works

- Try changing the technical indicators

- Experiment with different risk limits

- Test with different AI models (switch between Groq, OpenAI, Cerebras)

Intermediate Level

- Add more stocks to analyze

- Implement new technical indicators (VWAP, Fibonacci, etc.)

- Build a dashboard to visualize trades

- Add backtesting (test strategies on historical data)

- Implement position sizing strategies (Kelly Criterion, etc.)

Advanced Level

- Add sentiment analysis (news, Twitter, Reddit)

- Implement machine learning for pattern recognition

- Build multi-timeframe analysis (5min, 15min, 1hour charts)

- Add options trading capabilities

- Implement portfolio management across multiple stocks

The Bigger Picture: Why Agentic AI Matters

Building this trading agent taught me lessons that apply far beyond trading:

1. Simplicity Wins:

A well-designed single agent beats a complex multi-agent system. This applies to business processes, software architecture, and life.

2. Token Efficiency = Cost Efficiency:

Every API call costs money. In AI systems, efficiency isn’t just about speed — it’s about economics.

3. Model Agnostic = Future Proof:

Technology changes fast. Building systems that can swap components (like AI models) keeps you flexible.

4. Autonomous Doesn’t Mean Unsupervised:

The agent is autonomous, but it needs rules, constraints, and monitoring. True AI safety comes from good design, not just hoping the AI does the right thing.

5. Real-World AI is Messy:

Building this taught me about API rate limits, network timeouts, broker API quirks, and a hundred other practical issues you don’t face with toy examples.

Resources & Next Steps

If you want to build this yourself or learn more:

GitHub Repository:

https://github.com/marketcalls/Agentic-Trader

Prerequisites:**

- Basic Python knowledge

- Understanding of REST APIs

- Stock market basics (what’s a BUY/SELL order?)

- Basic terminal/command line skills

Documentation:

- OpenAI Agents SDK: https://openai.github.io/openai-agents-python/

- LiteLLM: https://docs.litellm.ai/

- OpenAlgo: https://openalgo.in/

- TA-Lib: https://ta-lib.org/

Community:

Join discussions about agentic AI and algo trading in Discord community. Share your learnings, ask questions, and help others.

Final Thoughts

Building an agentic trader is an incredible learning experience. You’ll learn about:

- AI agent architecture

- Real-time data processing

- Risk management

- API integration

- System design

- Financial markets

But remember: this is about learning, not about making quick money. The real value is in understanding how these systems work, so you can build other agentic systems (customer service bots, research assistants, automation tools) with the same principles.

The future is agentic. AI won’t just answer questions — it will take actions, make decisions, and accomplish goals autonomously. Understanding how to build, constrain, and optimize these systems is a valuable skill, whether you’re building trading agents, business automation, or the next AI startup.

Start simple, learn deeply, and build responsibly.

Happy building!

About This Article:

This article describes an open-source educational prototype. All code is available under MIT License. The author and contributors are not responsible for any financial losses. Trading involves substantial risk. Always test in paper trading mode first.

Want to Contribute?

The project is open source. Submit issues, pull requests, or feedback on GitHub.

Connect:

If you build something cool with this or have questions, share your journey. The best way to learn is by teaching others.

]]> Rajandran R open-source openai-agents-sdk ai-trading agentic-ai openalgo https://www.learningfromdata.zingg.ai/p/the-identity-crisis-why-getting-the https://www.learningfromdata.zingg.ai/p/the-identity-crisis-why-getting-the Thu, 23 Oct 2025 14:56:40 GMT

Matthew Niederberger, Founder of Martech Therapy recently wrote something that stopped me in my tracks:

“If you’re a practitioner still trying to get a unified customer ID working across channels, you feel left behind technologically and question your competence.”

This single sentence captures a crisis that’s quietly plaguing data engineering teams across the industry. While vendor booths showcase AI-driven personalization and predictive customer journeys, talented practitioners are privately wrestling with something supposedly “simpler”: building a unified customer ID that actually works.

The cruel irony? They’re doing the hardest, most important work. But in a market obsessed with bleeding-edge features, nobody’s celebrating it.

The Vendor Narrative Gap

So why do talented teams feel like they’re failing when they’re still working on identity?

Matthew Niederberger nails the dynamic: vendors showcase what’s possible, not what’s required to get there.

Vendor conference presentations follow a formula:

1. Demonstrate the amazing outcomes (real-time personalization! predictive journeys!)

2. Show the sleek UI that makes it look effortless

3. Mention “unified customer data” as an assumed prerequisite

4. Move quickly past the foundation to focus on the flashy features

The implicit message: if you’re still working on unified identity, you’re behind. Everyone else has already solved this.

But that’s not remotely true. The vast majority of organizations are still struggling with identity resolution. They’re just not talking about it publicly because it feels like admitting failure.

The result is a massive selection bias. We hear about the 5% who’ve moved on to advanced use cases. We don’t hear about the 95% still building foundations—not because they’re incompetent, but because the foundations are genuinely difficult to build well.

The Imposter Syndrome Epidemic

I’ve had countless conversations with data teams at conferences, in Slack channels, and on video calls. A pattern emerges again and again:

“We’re still working on identity resolution.”
“We haven’t gotten to the AI stuff yet.”
“We’re just trying to get our customer IDs right first.”

The word “just” does a lot of heavy lifting in those sentences. It diminishes foundational engineering work into a checkbox, a prerequisite you should have already mastered before moving on to the “real” work.

But here’s what nobody is saying loudly enough: unified identity isn’t a prerequisite. It’s the final boss of data engineering.

Why Unified Identity Is Actually Hard

Let’s be brutally honest about what “getting a unified customer ID working across channels” actually requires:

The Data Quality Reality

Real-world customer data is messy in ways that mock deterministic rules:

• Sarah Johnson gets married and becomes Sarah Martinez

• john.smith@gmail.com and jsmith@gmail.com might be the same person (or different people with similar names)

• Phone numbers get recycled—last year’s customer is this year’s prospect

• Addresses have hundreds of valid formatting variations

• People move, change jobs, update preferences

• Data entry errors create systematic inconsistencies

A unified customer ID system has to handle all of this, at scale, while maintaining high accuracy. Miss too many matches and your “360-degree customer view” is fragmented. Create false positives and you’re merging different people, violating privacy and destroying trust.

The Technical Challenge

You’re not matching records in a single clean database. You’re resolving identities across:

• Web analytics (cookie-based, increasingly unreliable)

• Mobile apps (device IDs, app-specific identifiers)

• CRM systems (email, phone, account IDs)

• Point-of-sale systems (loyalty cards, payment methods)

• Call center interactions (phone numbers, case IDs)

• Marketing platforms (each with their own identifier schemes)

• Third-party data sources (with their own quality issues)

Each source has different identifier types, different data quality standards, and different update frequencies. You’re not just joining tables—you’re probabilistically determining which records across fundamentally different systems represent the same human being.

The Scale Problem

Now multiply this across millions or billions of records. The naive approach—comparing every record to every other record—is computationally impossible. You need sophisticated blocking strategies, efficient algorithms, and infrastructure that can process massive datasets without collapsing under its own weight.

And you need to do this continuously, not as a one-time batch job. New data arrives constantly. Identities evolve. Your resolution system has to keep pace.

The Governance Complexity

Layer on consent management, privacy regulations, data retention policies, and the right-to-be-forgotten requirements. Your unified ID system isn’t just a technical challenge—it’s a governance framework that has to work across jurisdictions, respect customer preferences, and provide audit trails.

This isn’t “basic.” This is some of the most complex data engineering work you can do.

The Three-Layer Approach That Actually Works

So how do you actually build identity resolution that works at warehouse scale? Let me break down the architecture that successful implementations share:

Layer 1: Blocking

You can’t compare every record to every other record—that’s O(n²) complexity and computationally impossible at billion-record scale. Smart blocking reduces potential comparisons by 99%+ while maintaining high recall.

The key is choosing blocking keys that cast a wide enough net to catch true matches while being selective enough to make comparison tractable. This might mean blocking on:

• First three characters of last name + ZIP code

• Phone area code + first name

• Email domain + approximate birth date

• Phonetic encoding of names

Good blocking strategies use multiple passes with different keys, ensuring that records with typos or variations still get compared. At Zingg, we learn blocking directly on all the fields present in the data from the user data and labels to block different datasets efficiently.

Layer 2: Matching

This is where ML earns its keep. For each pair of records that passed blocking, you need to score the likelihood they represent the same entity.

Modern matching models use features like:

• String similarity (edit distance, Jaro-Winkler, token-based matching)

• Phonetic similarity (Soundex, Metaphone, NYSIIS)

• Behavioral patterns (purchase timing, channel preferences, engagement signals)

• Temporal proximity (how close in time were interactions?)

• Geographic consistency (do locations make sense together?)

The output isn’t binary—it’s probabilistic scoring. This pair has a 0.95 probability of being a match. That pair scores 0.45. This gives you flexibility in tuning precision vs. recall based on your use case.

Layer 3: Clustering

The final challenge: transform pairwise match scores into entity groups. This is where you resolve transitive relationships.

If Record A matches Record B with 0.92 confidence, and Record B matches Record C with 0.88 confidence, are A, B, and C all the same entity? What if A and C compared directly would only score 0.65?

Clustering algorithms handle these transitivity questions, creating stable entity groups even as new records arrive and scores shift over time. This layer also handles entity splits (when you discover you incorrectly merged two people) and entity merges (when new evidence shows records you thought were separate are actually one entity).

Why This Must Happen In the Warehouse And Datalake

Here’s a critical architectural decision: entity resolution must run natively in your data warehouse or data lake. Moving billions of records out for processing defeats the entire purpose of warehouse-native architecture.

Data Gravity

Your customer data is already in Snowflake, Databricks, BigQuery, or Redshift. That’s where your transformation pipelines run, where your analytics queries execute, where your data science models train. Extracting data to a separate entity resolution system creates:

• Data movement costs (bandwidth, storage, processing)

• Synchronization complexity (keeping multiple copies current)

• Latency (time to move data before resolution can happen)

• New data silos (exactly what warehouse consolidation was supposed to eliminate)

Incremental Processing

Entity resolution isn’t a one-time batch job. New customer records arrive constantly—from web signups, mobile registrations, purchase transactions, support interactions. You need incremental processing that resolves new records against existing entities without full dataset re-processing.

This only works efficiently when resolution runs where the data lives, using the warehouse’s native capabilities for incremental updates and change data capture.

Integration with Data Pipelines

Your entity resolution system needs to integrate seamlessly with:

• dbt transformations and data modeling

• Reverse ETL for audience activation

• BI tools for reporting and analytics

• ML platforms for model training

• Data quality frameworks for monitoring

• AI agents for context

This integration is natural when everything operates in the same environment. It becomes painful when entity resolution is a separate system with its own APIs, data formats, and operational requirements.

Governance and Security

Your data warehouse already has:

• Access controls and role-based permissions

• Audit logging for compliance

• Encryption at rest and in transit

• Data masking and tokenization capabilities

• Retention policies and deletion workflows

Replicating customer data to an external entity resolution system means duplicating all of this governance infrastructure. It becomes a privacy and compliance nightmare with GDPR, CCPA and other applicable laws. Run resolution in the warehouse, and you inherit these controls automatically.

The Human-in-the-Loop Problem

Here’s what the automation-obsessed narrative misses: pure ML automation for entity resolution rarely works in practice.

Why Full Automation Fails

Even the best ML models make mistakes on edge cases:

• Common names with similar addresses (are these two John Smiths in the same apartment building one person or two?)

• Shared email addresses (family members, assistants, corporate accounts)

• Identity changes (marriages, name corrections, gender transitions)

• Data entry errors that create systematic patterns the model learns incorrectly

• Inadequate data capture

You need human judgment on ambiguous cases, and you need to feed that judgment back into the model to improve over time.

Workflows That Actually Work

Successful implementations build workflows for:

Training data labeling: Which record pairs are matches, which are non-matches? Your initial model needs labeled examples, and getting these labels right determines everything downstream.

Threshold tuning: ML models output probability scores, but you choose the threshold. Score above 0.90 is an automatic match, below 0.40 is automatic non-match, but what about the 0.65 that could go either way? Human review helps calibrate these thresholds based on business impact.

Edge case review: The model flags cases it’s least confident about. A data steward reviews them, makes decisions, and those decisions become training examples for model tuning.

Continuous model refinement: Data patterns evolve. New data sources get added. Data quality issues emerge and get fixed. The model needs periodic retraining, and human feedback on recent predictions drives that retraining.

Dispute resolution: When downstream teams question why records were merged or split, you need workflows to investigate, explain the decision, and potentially override the model when it’s wrong.

The best systems don’t eliminate human review—they make it efficient. Instead of manually reviewing millions of records, you review the hundreds that matter most.

Common Implementation Mistakes

I’ve seen these patterns repeatedly as organizations tackle identity resolution:

Mistake 1: Starting Too Big

Teams try to resolve all entity types simultaneously—customers, accounts, products, locations, employees. This creates enormous scope, makes success criteria fuzzy, and delays time to value.

Better approach: Start with one high-value entity type (usually customers). Prove the value, build organizational capability, then expand to other entity types. Each new entity type gets easier because you’ve learned the patterns.

Mistake 2: Ignoring Data Quality

Entity resolution surfaces every data quality issue you’ve been ignoring. If 40% of your email addresses are null or invalid, you can’t rely on email matching. If name fields contain inconsistent formatting, parsing errors, or non-name data, string matching becomes unreliable.

Better approach: Treat entity resolution and data quality as inseparable workstreams. Profile your data first. Fix systematic quality issues before expecting resolution to work well. Use entity resolution results to identify and prioritize remaining quality problems.

Mistake 3: Treating It as a Project

Organizations approach entity resolution as a one-time project: scope it, build it, deploy it, move on. Then they’re shocked when accuracy degrades over time as data patterns shift.

Better approach: Treat entity resolution as infrastructure that requires ongoing ownership, monitoring, and refinement—just like your data pipelines or ML models. Assign clear ownership, establish SLAs, build monitoring dashboards, schedule regular review.

Mistake 4: Optimizing for Perfection

Teams spend months trying to achieve 99.9% accuracy before releasing anything. Meanwhile, downstream teams make decisions based on no identity resolution at all (effectively 0% accuracy).

Better approach: Ship at 95% accuracy, deliver value, iterate to higher accuracy based on real-world feedback. You learn more from production usage in two weeks than from six months of pre-release testing. Start good, iterate to great.

Mistake 5: Building From Scratch

Organizations underestimate the complexity and decide to build custom entity resolution systems. Eighteen months later, they have something that works on small datasets but can’t scale, has accuracy issues they don’t understand, and requires specialized knowledge that only one person has.

Better approach: Use proven solutions—whether open source frameworks or commercial platforms—that solve the hard algorithmic and scaling problems. Invest your engineering effort in customization, integration, and business-specific logic, not reinventing core entity resolution algorithms.

The Real Question

At this point in conversations, someone usually asks: “Should we do ML-based entity resolution?”

That’s the wrong question.

The real question is: “Can our organization afford to build composable architectures on a foundation of deterministic rules that we know will fail at scale?”

Every organization moving to warehouse-native architectures will eventually face the identity resolution challenge. The only question is whether you address it proactively—as part of your architectural planning—or reactively, after your first major data quality incident.

The reactive path looks like this:

• Build your composable architecture on simple deduplication rules

• Activate audiences and launch campaigns

• Discover that 15% of your “unified” customers are actually duplicates

• Or worse, discover you’ve merged different people and violated privacy

• Scramble to retrofit proper identity resolution

• Rebuild audiences and downstream dependencies

• Deal with the business impact and loss of trust

The proactive path looks like this:

• Recognize identity resolution as foundational, not an afterthought

• Build or adopt ML-based entity resolution that runs in your warehouse

• Start with one entity type, prove value, iterate

• Build downstream capabilities on accurate identity from day one

• Scale confidently knowing your foundation is solid

The reactive path is more common (because people underestimate the challenge). The proactive path is more successful (because foundations matter).

What Actually Endures

Here’s the uncomfortable truth about chasing bleeding-edge features without solid foundations:

A CDP that can’t accurately unify customer IDs is just an expensive data warehouse with a marketing UI.

You can bolt on AI-driven orchestration, predictive segmentation, and real-time decisioning. But if your underlying identity graph is wrong—if you think one customer is three people, or three people are one customer—every downstream capability inherits those errors.

Your AI doesn’t fix bad identity data. It amplifies it. Your personalization engine delivers fragmented experiences. Your predictive models train on fictional customer journeys. Your real-time decisioning makes decisions based on incomplete context.

The organizations actually positioned to leverage advanced CDP capabilities aren’t the ones with the longest feature lists. They’re the ones who took the time to build reliable identity resolution first.

The Real Achievement

Let me share two examples of organizations that built this foundation right.

Fortnum & Mason, the 300-year-old luxury British retailer, faced a classic challenge: customer data fragmented across restaurant bookings, email signups, online transactions, and in-store purchases. They initially tried a third-party service for identity resolution, but it created non-persistent identifiers, offered limited visibility into the matching process, and raised privacy concerns about sharing their entire dataset externally.

Jon Moss, Fortnum’s Customer Engagement Director, describes the transformation:

“For the first time, we’re able to understand how customers are shopping with us—online, in-store, over the phone, or in restaurants. Zingg has helped us unify this data and gain insights we never had before.”

By implementing ML-based entity resolution running natively in their data warehouse, Fortnum & Mason built:

• Persistent, unified customer identifiers across all touchpoints

• Complete visibility and control over the matching process

• Privacy-compliant resolution that keeps sensitive data in their own infrastructure

• A foundation for their composable CDP architecture

Orthodox Union, one of the largest Orthodox Jewish organizations in the US, operates over 40 websites, 5 mobile applications, and custom CRMs for different departments. Shelomo Dobkin, their Director of Product Development, explains:

“We moved from using Zingg Open Source to the Enterprise version on Snowflake, making it the engine powering our golden records. Zingg’s powerful features and clean results have really set us apart.”

Their implementation required understanding not just individual identities but household relationships—a critical requirement for their community-focused mission. They built entity resolution that could:

• Match people across multiple websites and CRM systems

• Understand household connections and family relationships

• Create golden records that serve as the foundation for all downstream systems

• Process data incrementally as new interactions occur across their digital properties

What makes these achievements remarkable isn’t just the technical implementation. It’s what they enable. This is world-class data engineering that delivers measurable business value:

• Marketing teams can trust their audiences

• Analytics teams can build on reliable customer journeys

• Data science teams can train models on accurate historical data

• Operations teams can deliver consistent experiences across channels

• Legal and compliance teams can demonstrate proper governance

Everything else—the AI features, the real-time orchestration, the predictive models—becomes possible because the foundation is solid.

Celebrating Foundational Work

The industry needs to reframe how we talk about identity resolution and other foundational data work.

These aren’t prerequisites that should be quick and easy. They’re achievements that deserve recognition.

When a team successfully builds unified customer IDs across disparate systems, that’s a case study worth sharing. When an organization finally has audiences their teams trust, that’s an accomplishment worth celebrating. When identity resolution maintains accuracy as data scales from millions to billions of records, that’s engineering excellence.

The castle that survives wave after wave isn’t the tallest or flashiest. It’s the one with foundations deep enough to withstand the tide.

Moving Forward

If you’re a data practitioner still working on unified customer identity, you’re not behind. You’re doing the work that matters most.

Don’t let vendor demos and curated LinkedIn posts make you feel like an imposter. The bleeding-edge use cases get all the attention precisely because they’re rare. The foundational work gets less visibility precisely because it’s what everyone is actually struggling with.

Focus on making your identity infrastructure trustworthy and durable:

• Invest in ML-based entity resolution that can handle real-world data messiness

• Build in your data warehouse and datalake so you’re not creating new data silos

• Design for continuous operation, not one-time batch processing

• Create human-in-the-loop workflows for edge cases and model refinement

• Treat it as infrastructure that requires ongoing ownership and monitoring

The teams that master this foundation aren’t behind. They’re building something that will outlast the next wave of features, the next vendor pivot, the next technological hype cycle.

That’s not just the basics. That’s the real work. And it’s work worth doing right.

---

What’s your experience building unified customer identity? Where did you hit the biggest obstacles? I’d love to hear your stories—the successes and the struggles.

]]> Sonal Goyal https://mrkaran.dev/posts/ai-home-cooked-software/ https://mrkaran.dev/posts/ai-home-cooked-software/ Sun, 05 Oct 2025 00:00:00 GMT Everyone is worried that AI will replace programmers. They’re missing the real revolution: AI is turning everyone into one.

I’ve been noticing a new pattern: people with deep domain knowledge but no coding experience are now building their own tools. Armed with AI assistants, they can create custom workflows in a matter of days, bypassing traditional development cycles. Are these solutions production-ready? Not even close. But they solve urgent, specific problems, and that’s what matters. Tasks that once required weeks of specialized training are quickly becoming weekend projects.

This trend is happening even within the AI companies themselves. Anthropic, for example, shared how their own teams use Claude to accelerate their work. Crucially, this isn’t limited to developers. Their post details how non-technical staff now build their own solutions and create custom automations, providing a powerful real-world example of this new paradigm.

Home-Cooked Software#

Why search for a generic tool when you can build exactly what you need? This question leads to what I call ‘home-cooked software’: small, personal applications we build for ourselves, tailored to our specific needs. Robin Sloan beautifully describes building an app as making “a home-cooked meal,” while Maggie Appleton writes about “barefoot developers” creating software outside traditional industry structures.

What’s new isn’t the concept but the speed and accessibility. With AI, a custom export format, a specific workflow, or the perfect integration is now an afternoon’s work. We’re entering an unprecedented era where the barrier between wanting a tool and having it has nearly vanished.

But let’s be clear: the journey from a prototype to a production-ready application is as challenging as ever. In my experience, an AI can churn out a first draft in a few hours, which gets you surprisingly far. But the devil is in the details, and the last stretch of the journey – handling edge cases, ensuring security, and debugging subtle issues – can stretch into weeks. This distinction is crucial. AI isn’t replacing programmers; it’s creating millions of people who can build simple tools. There’s a significant difference.

The New Economics#

AI is fundamentally reshaping the economics of building software. Before AI, even a simple tool required a significant time investment in learning programming basics, understanding frameworks, and debugging. Only tools with broad appeal or critical importance justified the effort. Now, that effort is measured in hours, not months, and the primary barrier is no longer technical knowledge, but imagination and a clear understanding of one’s own needs.

This doesn’t apply to complex or security-critical systems, where deep expertise remains essential. But for the long tail of personal utilities, automation scripts, and custom workflows, the math has changed completely. I’m talking about solving all those minor irritations that pile up: the script to reformat a specific CSV export, the dashboard showing exactly the three metrics you care about, or a script that pulls data from a personal project management tool to sync with an obscure time-tracking app.

These tools might be held together with digital duct tape, but they solve real problems for real people. And increasingly, that’s all that matters.

The Hidden Costs#

But this newfound capability isn’t free. It comes with what I call the “AI Tax”: a set of hidden costs that are rarely discussed.

First, prompt engineering can be surprisingly time-consuming, especially for tasks of moderate complexity. While simple requests are often straightforward, anything more nuanced can become an iterative dialogue. You prompt, the AI generates a flawed output, you clarify the requirements, and it returns a new version that misses a different detail. It’s a classic 80/20 scenario: you get 80% of the way there with a simple prompt, but achieving the final 20% of correctness requires a disproportionate amount of effort in refining, correcting, and clarifying your intent to the model.

Second, there’s the verification burden. Every line of AI-generated code is a plausible-looking liability. It may pass basic tests, only to fail spectacularly in production with an edge case you never considered. AI learned from the public internet, which means it absorbed all the bad code along with the good. SQL injection vulnerabilities, hardcoded secrets, race conditions—an AI will happily generate them all with complete confidence.

Perhaps the most frustrating aspect is “hallucination debugging”: the uniquely modern challenge of troubleshooting plausible-looking code that relies on APIs or methods that simply don’t exist. Your codebase becomes a patchwork of different AI-generated styles and patterns. Six months later, it’s an archaeological exercise to determine which parts you wrote and which parts an AI contributed.

But the most significant danger is that AI enables you to build systems you don’t fundamentally understand. When that system inevitably breaks, you lack the foundational knowledge to debug it effectively.

Building for One#

Despite these challenges, there’s something profoundly liberating about building software just for yourself. Instead of just sketching out ideas, I’ve started building these small, specific tools. For this blog, I wanted a simple lightbox for images; instead of pulling in a heavy external library, I had Claude write a 50-line JavaScript snippet that did exactly what I needed. I built a simple, single-page compound interest calculator tailored for my own financial planning. To save myself from boilerplate at work, I created prom2grafana, a tool that uses an LLM to convert Prometheus metrics into Grafana dashboards.

Ten years ago, I might have thought about generalizing these tools, making them useful for others, perhaps even starting an open source project. Today? I just want a tool that works exactly how I think. I don’t need to handle anyone else’s edge cases or preferences. Home-cooked software doesn’t need product-market fit—it just needs to fit you.

We’re witnessing the emergence of a new software layer. At the base are the professionally-built, robust systems that power our world: databases, operating systems, and rock-solid frameworks. In the middle are commercial applications built for broad audiences. And at the top, a new layer is forming: millions of tiny, personal tools that solve individual problems in highly specific ways.

This top layer is messy, fragile, and often incomprehensible to anyone but its creator. It’s also incredibly empowering. Creating simple software is becoming as accessible as writing. And just as most writing isn’t professional literature, most of this new software won’t be professional-grade. That’s not just okay; it’s the point.

The implications are profound. Subject-matter experts can now solve their own problems without waiting for engineering resources, and tools can be hyper-personalized to a degree that is impossible for commercial software. This unlocks a wave of creativity, completely unconstrained by the need to generalize or find a market.

Yes, there are legitimate concerns. Security is a real risk, though the profile changes when a tool runs locally on personal data with no external access. We’re creating personal technical debt, but when a personal tool breaks, the owner is the only one affected. They can choose to fix it, rebuild it, or abandon it without impacting anyone else. Organizations, on the other hand, will soon have to grapple with the proliferation of incompatible personal tools and establish new patterns for managing them.

But these challenges pale in comparison to the opportunities. The barrier between user and creator is dissolving. We’re entering the age of home-cooked software, where building your own tool is becoming as natural as cooking your own meal.

The kitchen is open. What will you cook?

]]> Karan Sharma https://mrkaran.dev/posts/state-homelab-2025/ https://mrkaran.dev/posts/state-homelab-2025/ Sat, 04 Oct 2025 00:00:00 GMT /config.sh file. This file can also contain pre_deploy and post_deploy hooks for custom actions. The justfile makes daily operations trivial: # Deploy a single stack to a machine just deploy-stack floyd-homelab-1 immich # View the logs for a stack just logs floyd-homelab-1 immich # Test a deployment without making changes just dry-run floyd-homelab-1 This system provides fine-grained control over deployments, with support for actions like up, down, restart, pull, and recreate (which also removes persistent volumes). Container & Configuration Patterns# To keep the system consistent, I follow a few key patterns: Data Persistence: Instead of using Docker named volumes, I use host bind mounts. All persistent data for a service is stored in a dedicated directory on the host, typically /data/. This makes backups and data management more transparent. Reverse Proxy Network: The Caddy stack defines a shared Docker network called public_proxy. Other stacks that need to be exposed to the internet are configured to join this network. This allows Caddy to discover and proxy them without exposing their ports on the host machine. I have written about this pattern in detail in a previous post. Port Exposure: Services behind the reverse proxy use the expose directive in their docker-compose.yml to make ports available to Caddy within the Docker network. I avoid binding ports directly with ports unless absolutely necessary. Multi-Machine Topology# The homelab comprises three distinct machines to provide isolation and redundancy. floyd-homelab-1 (Primary Server): The core of the homelab, running on the AMD hardware detailed above. It runs data-intensive personal services (e.g., Immich, Paperless-ngx) and is accessible only via the Tailscale network. floyd-pub-1 (Public VPS): A small cloud VPS that hosts public-facing services requiring high availability, such as DNS utilities, analytics, and notification relays. floyd-monitor-public (Monitoring VPS): A small Hetzner VM running Gatus for health checks. Its independence ensures that I am alerted if the primary homelab or home network goes offline. This distributed setup isolates my home network from the public internet and ensures that critical public services remain online even if the home server is down for maintenance. Hosted Services# The following is a breakdown of the services, or “stacks,” running on each machine. A few key services that are central to the homelab are detailed further in the next section. floyd-homelab-1 (Primary Server)# Actual: A local-first personal finance and budgeting tool. Caddy: A powerful, enterprise-ready, open source web server with automatic HTTPS. Gitea: A Git service for personal projects. Glance: A dashboard for viewing all my feeds and data in one place. Immich: A photo and video backup solution, directly from my mobile phone. Karakeep: An app for bookmarking everything, with AI-based tagging and full-text search. Owntracks: A private location tracker for recording my own location data. Paperless-ngx: A document management system that transforms physical documents into a searchable online archive. Silverbullet: A Markdown-based knowledge management and note-taking tool. floyd-monitor-public (Monitoring VPS)# Caddy: Reverse proxy for the services on this node. floyd-pub-1 (Public VPS)# Beszel-agent: The agent for the Beszel monitoring platform. Caddy: Reverse proxy for the services on this node. Cloak: A service to securely share sensitive text with others. Doggo: A command-line DNS Client for Humans, written in Golang. Ntfy: A self-hosted push notification service. prom2grafana: A tool to convert Prometheus metrics to Grafana dashboards and alert rules using AI. Umami: A simple, fast, privacy-focused alternative to Google Analytics. Service Highlights# Technitium: A Powerful DNS Server# I came across Technitium DNS after seeing a recommendation from @oddtazz, and it has been a revelation. For anyone who wants more than just basic ad blocking from their DNS server, it’s a game-changer. It serves as both a recursive and authoritative server, meaning I don’t need a separate tool like unbound to resolve from root hints. The level of configuration is incredible—from DNSSEC, custom zones, and SOA records to fine-grained caching control. The UI is a bit dated, but that’s a minor point for me given the raw power it provides. It is a vastly underrated tool for any homelabber who wants to go beyond Pi-hole or AdGuard Home. Beszel: Lightweight Monitoring# For a long time, I felt that monitoring a homelab meant spinning up a full Prometheus and Grafana stack. Beszel is the perfect antidote to that complexity. It provides exactly what I need for basic node monitoring—CPU, memory, disk, and network usage—in a simple, lightweight package. It’s incredibly easy to set up and provides a clean, real-time view of my servers without the overhead of a more complex system. For a simple homelab monitoring setup, it’s hard to beat. Gatus: External Health Checks# While Beszel monitors the servers from the inside, Gatus watches them from the outside. Running on an independent Hetzner VM, its job is to ensure my services are reachable from the public internet. It validates HTTP status codes, response times, and more. This separation is crucial; if my entire home network goes down, Gatus is still online to send an alert to my phone. It’s the final piece of the puzzle for robust monitoring, ensuring I know when things are broken even if the monitoring service itself is part of the outage. Storage and Backup Strategy# Data integrity and recoverability are critical. My strategy is built on layers of redundancy and encryption. Storage: BTRFS RAID 1 + LUKS Encryption# I chose BTRFS for its modern features: Checksumming: Protects against silent data corruption. Copy-on-Write: Enables instantaneous, low-cost snapshots. Transparent Compression: zstd compression saves space without significant performance overhead. The two 4TB drives are mirrored in a RAID 1 array, providing redundancy against a single drive failure. The entire array is encrypted using LUKS2, with the key stored on the boot SSD for automatic mounting. This protects data at rest in case of physical theft or drive disposal. Mount options in /etc/fstab: /dev/mapper/crypt-sda /mnt/storage btrfs defaults,noatime,compress=zstd 0 2 Backup: Restic + Cloudflare R2# RAID does not protect against accidental deletion, file corruption, or catastrophic failure. My backup strategy follows the 3-2-1 rule. Daily, automated backups are managed by systemd timers running restic. Backups are encrypted and sent to Cloudflare R2, providing an off-site copy. R2 was chosen for its zero-cost egress, which is a significant advantage for restores. The backup script covers critical application data and the Docker Compose configurations: BACKUP_PATHS=( "/mnt/storage" # All application data "/home/karan/stacks" # Docker Compose configs ) Each backup run reports its status to a healthchecks.io endpoint, which sends a push notification on failure. I must appreciate its generous free tier, which is more than sufficient for my needs. Conclusion# This homelab represents a shift in philosophy from exploring complexity to valuing simplicity and reliability. The upfront hardware investment of ~$1,200 is offset by eliminating recurring cloud hosting costs and providing complete control over my data and services. For those considering a homelab, my primary recommendation is to start with a simple, well-understood foundation. A reliable machine with a solid backup strategy is more valuable than a complex, hard-to-maintain cluster. The goal is to build a system that serves your needs, not one that you serve.]]> Introduction#

For the past five years, I have maintained a homelab in various configurations. This journey has served as a practical exploration of different technologies, from Raspberry Pi clusters running K3s to a hybrid cloud setup and eventually a cloud-based Nomad setup. Each iteration provided valuable lessons, consistently highlighting the operational benefits of simplicity.

This article details the current state of my homelab. A primary motivation for this build was to dip my toes into “actual” homelabbing—that is, maintaining a physical server at home. The main design goal was to build a dedicated, reliable, and performant server that is easy to maintain. This led me to move away from complex container orchestrators like Kubernetes in favor of a more straightforward Docker Compose workflow. I will cover the hardware build, software architecture, and the rationale behind the key decisions.

Hardware Configuration#

After considerable research, I selected components to balance performance, power efficiency, and cost. The server is designed for 24/7 operation in a home environment, making noise and power consumption important considerations.

The Build#

Component	Choice	Price
CPU	AMD Ryzen 5 7600X (6-core, 4.7 GHz)	$167.58
CPU Cooler	ARCTIC Liquid Freezer III Pro 360	$89.99
Motherboard	MSI B650M Gaming Plus WiFi	$225.83
RAM	Kingston FURY Beast 32GB DDR5-6000	$136.99
Boot Drive	WD Blue SN580 500GB NVMe	$88.76
Storage 1	WD Red Plus 4TB (5400 RPM)	$99.99
Storage 2	Seagate IronWolf Pro 4TB (7200 RPM)	$150.00
Case	ASUS Prime AP201 MicroATX	$89.99
PSU	Corsair SF750 (80+ Platinum)	$169.99
Total		$1,219.12

Component Rationale#

• CPU: The Ryzen 5 7600X provides a strong price-to-performance ratio. Its 6 cores offer ample headroom for concurrent containerized workloads and future experimentation.
• Storage: The boot drive is a 500GB NVMe for fast OS and application performance. The primary storage consists of two 4TB HDDs in a BTRFS RAID 1 configuration. To mitigate the risk of correlated failures, I chose drives from different manufacturers (WD and Seagate) purchased at different times.
• RAM: 32GB of DDR5-6000 provides sufficient memory for a growing number of services without risking contention.
• Case & PSU: The ASUS Prime AP201 is a compact MicroATX case with a clean aesthetic suitable for a home office. The Corsair SF750 (80+ Platinum) PSU was chosen for its efficiency and to provide capacity for a future GPU for local LLM or transcoding workloads.

System Architecture & Deployment#

My previous setups involved Kubernetes and Nomad, but the operational overhead proved unnecessary for my use case. I have since standardized on a Git-based, Docker Compose workflow that prioritizes simplicity and transparency.

Directory Structure and “Stacks”#

The core of the system is a Git repository that holds all configurations. Each service is defined as a self-contained “stack” in its own directory. The structure is organized by machine, making it easy to manage multiple environments:

homelab/
├── deploy.sh                 # Main deployment script
├── justfile                  # Task runner for common commands
└── machines/
    ├── floyd-homelab-1/      # Primary home server
    │   ├── config.sh         # SSH and deployment settings
    │   └── stacks/
    │       ├── immich/
    │       │   └── docker-compose.yml
    │       └── paperless/
    │           └── docker-compose.yml
    └── floyd-pub-1/          # Public-facing VPS
        ├── config.sh
        └── stacks/
            ├── caddy/
            └── ntfy/

This modular approach allows me to manage each application’s configuration, including its docker-compose.yml and any related files, as an independent unit.

Deployment Workflow#

Deployments are handled by a custom deploy.sh script, with a justfile providing a convenient command-runner interface. The process is fundamentally simple:

1. Sync: rsync copies the specified stack’s directory from the local Git repository to a REMOTE_BASE_PATH (e.g., /opt/homelab) on the target machine.
2. Execute: ssh runs the appropriate docker compose command on the remote machine.

Each machine’s connection settings (SSH_HOST, SSH_USER, REMOTE_BASE_PATH) are defined in its machines/<name>/config.sh file. This file can also contain pre_deploy and post_deploy hooks for custom actions.

The justfile makes daily operations trivial:

# Deploy a single stack to a machine
just deploy-stack floyd-homelab-1 immich

# View the logs for a stack
just logs floyd-homelab-1 immich

# Test a deployment without making changes
just dry-run floyd-homelab-1

This system provides fine-grained control over deployments, with support for actions like up, down, restart, pull, and recreate (which also removes persistent volumes).

Container & Configuration Patterns#

To keep the system consistent, I follow a few key patterns:

• Data Persistence: Instead of using Docker named volumes, I use host bind mounts. All persistent data for a service is stored in a dedicated directory on the host, typically /data/<service-name>. This makes backups and data management more transparent.
• Reverse Proxy Network: The Caddy stack defines a shared Docker network called public_proxy. Other stacks that need to be exposed to the internet are configured to join this network. This allows Caddy to discover and proxy them without exposing their ports on the host machine. I have written about this pattern in detail in a previous post.
• Port Exposure: Services behind the reverse proxy use the expose directive in their docker-compose.yml to make ports available to Caddy within the Docker network. I avoid binding ports directly with ports unless absolutely necessary.

Multi-Machine Topology#

The homelab comprises three distinct machines to provide isolation and redundancy.

• floyd-homelab-1 (Primary Server): The core of the homelab, running on the AMD hardware detailed above. It runs data-intensive personal services (e.g., Immich, Paperless-ngx) and is accessible only via the Tailscale network.
• floyd-pub-1 (Public VPS): A small cloud VPS that hosts public-facing services requiring high availability, such as DNS utilities, analytics, and notification relays.
• floyd-monitor-public (Monitoring VPS): A small Hetzner VM running Gatus for health checks. Its independence ensures that I am alerted if the primary homelab or home network goes offline.

This distributed setup isolates my home network from the public internet and ensures that critical public services remain online even if the home server is down for maintenance.

Hosted Services#

The following is a breakdown of the services, or “stacks,” running on each machine. A few key services that are central to the homelab are detailed further in the next section.

floyd-homelab-1 (Primary Server)#

• Actual: A local-first personal finance and budgeting tool.
• Caddy: A powerful, enterprise-ready, open source web server with automatic HTTPS.
• Gitea: A Git service for personal projects.
• Glance: A dashboard for viewing all my feeds and data in one place.
• Immich: A photo and video backup solution, directly from my mobile phone.
• Karakeep: An app for bookmarking everything, with AI-based tagging and full-text search.
• Owntracks: A private location tracker for recording my own location data.
• Paperless-ngx: A document management system that transforms physical documents into a searchable online archive.
• Silverbullet: A Markdown-based knowledge management and note-taking tool.

floyd-monitor-public (Monitoring VPS)#

• Caddy: Reverse proxy for the services on this node.

floyd-pub-1 (Public VPS)#

• Beszel-agent: The agent for the Beszel monitoring platform.
• Caddy: Reverse proxy for the services on this node.
• Cloak: A service to securely share sensitive text with others.
• Doggo: A command-line DNS Client for Humans, written in Golang.
• Ntfy: A self-hosted push notification service.
• prom2grafana: A tool to convert Prometheus metrics to Grafana dashboards and alert rules using AI.
• Umami: A simple, fast, privacy-focused alternative to Google Analytics.

Service Highlights#

Technitium: A Powerful DNS Server#

I came across Technitium DNS after seeing a recommendation from @oddtazz, and it has been a revelation. For anyone who wants more than just basic ad blocking from their DNS server, it’s a game-changer. It serves as both a recursive and authoritative server, meaning I don’t need a separate tool like unbound to resolve from root hints. The level of configuration is incredible—from DNSSEC, custom zones, and SOA records to fine-grained caching control.

The UI is a bit dated, but that’s a minor point for me given the raw power it provides. It is a vastly underrated tool for any homelabber who wants to go beyond Pi-hole or AdGuard Home.

Beszel: Lightweight Monitoring#

For a long time, I felt that monitoring a homelab meant spinning up a full Prometheus and Grafana stack. Beszel is the perfect antidote to that complexity. It provides exactly what I need for basic node monitoring—CPU, memory, disk, and network usage—in a simple, lightweight package.

It’s incredibly easy to set up and provides a clean, real-time view of my servers without the overhead of a more complex system. For a simple homelab monitoring setup, it’s hard to beat.

Gatus: External Health Checks#

While Beszel monitors the servers from the inside, Gatus watches them from the outside. Running on an independent Hetzner VM, its job is to ensure my services are reachable from the public internet. It validates HTTP status codes, response times, and more.

This separation is crucial; if my entire home network goes down, Gatus is still online to send an alert to my phone. It’s the final piece of the puzzle for robust monitoring, ensuring I know when things are broken even if the monitoring service itself is part of the outage.

Storage and Backup Strategy#

Data integrity and recoverability are critical. My strategy is built on layers of redundancy and encryption.

Storage: BTRFS RAID 1 + LUKS Encryption#

I chose BTRFS for its modern features:

• Checksumming: Protects against silent data corruption.
• Copy-on-Write: Enables instantaneous, low-cost snapshots.
• Transparent Compression: zstd compression saves space without significant performance overhead.

The two 4TB drives are mirrored in a RAID 1 array, providing redundancy against a single drive failure. The entire array is encrypted using LUKS2, with the key stored on the boot SSD for automatic mounting. This protects data at rest in case of physical theft or drive disposal.

Mount options in /etc/fstab:

/dev/mapper/crypt-sda /mnt/storage btrfs defaults,noatime,compress=zstd 0 2

Backup: Restic + Cloudflare R2#

RAID does not protect against accidental deletion, file corruption, or catastrophic failure. My backup strategy follows the 3-2-1 rule.

Daily, automated backups are managed by systemd timers running restic. Backups are encrypted and sent to Cloudflare R2, providing an off-site copy. R2 was chosen for its zero-cost egress, which is a significant advantage for restores.

The backup script covers critical application data and the Docker Compose configurations:

BACKUP_PATHS=(
    "/mnt/storage"        # All application data
    "/home/karan/stacks"  # Docker Compose configs
)

Each backup run reports its status to a healthchecks.io endpoint, which sends a push notification on failure. I must appreciate its generous free tier, which is more than sufficient for my needs.

Conclusion#

This homelab represents a shift in philosophy from exploring complexity to valuing simplicity and reliability. The upfront hardware investment of ~$1,200 is offset by eliminating recurring cloud hosting costs and providing complete control over my data and services.

For those considering a homelab, my primary recommendation is to start with a simple, well-understood foundation. A reliable machine with a solid backup strategy is more valuable than a complex, hard-to-maintain cluster. The goal is to build a system that serves your needs, not one that you serve.

---

]]> Karan Sharma https://programmerlife1.wordpress.com/2025/10/01/porting-a-python-program-with-help-of-chat-gpt-1/ https://programmerlife1.wordpress.com/2025/10/01/porting-a-python-program-with-help-of-chat-gpt-1/ Wed, 01 Oct 2025 12:27:59 GMT Hi all,

In 2024, I have started porting a python library open tamil

https://github.com/Ezhil-Language-Foundation/open-tamil/.

I faced difficulties on tracking the port status of the application and noting the python symbols with php implementation with testing and documenting the changes over time because the code base itself not a small one.

so i have decided to prioritize tracking than the porting.

kick started the tracking with a mapping file and a php script fine tuned the process with explaining the difficulties

Step 1 :

i want a porting status tracker in php which reads the symbols in a python code and it require equivalent symbol with working status and documentation is it possible?

CHATGPT : Yes, it’s absolutely possible
You can build a porting status tracker in PHP that analyzes Python code, extracts symbols (functions, classes, methods, variables), and then checks against a mapping of equivalent PHP symbols with their status (e.g., ported / not ported / partially working) and optional documentation.

Here’s how you can approach it:

---

High-Level Approach

1. Parse Python code
   • Extract symbols like def function_name, class ClassName, global variables, and imports.
   • You can do this with regex (simpler) or by calling Python’s ast module and returning JSON for PHP to read.
2. Maintain a Mapping Table
   • A JSON or database table that maps:
     • python_symbol → php_equivalent, status, notes
   Example: { "print": { "php_equivalent": "echo", "status": "ported", "notes": "Direct replacement" }, "len": { "php_equivalent": "count", "status": "ported", "notes": "Works on arrays" }, "dict": { "php_equivalent": "associative array", "status": "in-progress", "notes": "Limited support" } }
3. Tracker Logic
   • For each symbol found in the Python code, check if it exists in the mapping.
   • Output a report (HTML table / JSON / CLI output) with:
     • Symbol name
     • PHP equivalent
     • Porting status
     • Documentation/Notes
4. Visualization
   • You can show progress in percentage: Total Symbols: 50 Ported: 30 (60%) Pending: 20 (40%)

---

Example PHP Implementation (Simple Version)

<?php

class PortingTracker {
    private $mapping;

    public function __construct($mappingFile) {
        $this->mapping = json_decode(file_get_contents($mappingFile), true);
    }

    public function analyzePythonFile($pythonFile) {
        $code = file_get_contents($pythonFile);

        // Simple regex to find defs and classes
        preg_match_all('/def\s+([a-zA-Z0-9_]+)/', $code, $functions);
        preg_match_all('/class\s+([a-zA-Z0-9_]+)/', $code, $classes);

        $symbols = array_merge($functions[1], $classes[1]);

        $report = [];
        foreach ($symbols as $symbol) {
            if (isset($this->mapping[$symbol])) {
                $report[$symbol] = $this->mapping[$symbol];
            } else {
                $report[$symbol] = [
                    "php_equivalent" => null,
                    "status" => "missing",
                    "notes" => "No mapping found"
                ];
            }
        }

        return $report;
    }
}

// Example usage
$tracker = new PortingTracker("mapping.json");
$result = $tracker->analyzePythonFile("example.py");

echo "<table border='1' cellpadding='5'>";
echo "<tr><th>Python Symbol</th><th>PHP Equivalent</th><th>Status</th><th>Notes</th></tr>";
foreach ($result as $py => $info) {
    echo "<tr>
            <td>{$py}</td>
            <td>{$info['php_equivalent']}</td>
            <td>{$info['status']}</td>
            <td>{$info['notes']}</td>
          </tr>";
}
echo "</table>";

---

Output Example (HTML Table)

Python Symbol	PHP Equivalent	Status	Notes
print	echo	ported	Direct replacement
len	count	ported	Works on arrays
dict	associative array	in-progress	Limited support
MyClass	NULL	missing	No mapping found

---

Next-Level Features

• Use Python AST (ast module) to extract symbols more reliably instead of regex.
  (You can run python3 script.py example.py > symbols.json and then load that JSON in PHP.)
• Add progress bar UI for tracking completion.
• Add search & filter for large projects.
• Store mapping in MySQL/Postgres for collaborative tracking.

---

Do you want me to make a standalone PHP tool (self-contained with regex parsing), or do you prefer a hybrid approach (use Python AST to export JSON, then PHP reads it)?
The hybrid one is much more reliable for real-world Python code.

This answer make me a decision on the porting of the python.

Started implementing the same but I have to satisfy the how ? inside me.

]]> Hariharan Uncategorized chatgpt php porting python https://workdone0.substack.com/p/binary-or-linear-the-hidden-math https://workdone0.substack.com/p/binary-or-linear-the-hidden-math Mon, 29 Sep 2025 14:58:34 GMT Shubham Kumar https://ravidwivedi.in/posts/singapore-trip/ https://ravidwivedi.in/posts/singapore-trip/ Tue, 23 Sep 2025 11:35:26 GMT In December 2024, I went on a trip through four countries - Singapore, Malaysia, Brunei, and Vietnam - with my friend Badri. This post covers our experiences in Singapore.

I took an IndiGo flight from Delhi to Singapore, with a layover in Chennai. At the Chennai airport, I was joined by Badri. We had an early morning flight from Chennai that would land in Singapore in the afternoon. Within 48 hours of our scheduled arrival in Singapore, we submitted an arrival card online. At immigration, we simply needed to scan our passports at the gates, which opened automatically to let us through, and then give our address to an official nearby. The process was quick and smooth, but it unfortunately meant that we didn’t get our passports stamped by Singapore.

Before I left the airport, I wanted to visit the nature-themed park with a fountain I saw in pictures online. It is called Jewel Changi, and it took quite some walking to get there. After reaching the park, we saw a fountain that could be seen from all the levels. We roamed around for a couple of hours, then proceeded to the airport metro station to get to our hotel.

There were four ATMs on the way to the metro station, but none of them provided us with any cash. This was the first country (outside India, of course!) where my card didn’t work at ATMs.

To use the metro, one can tap the EZ-Link card or bank cards at the AFC gates to get in. You cannot buy tickets using cash. Before boarding the metro, I used my credit card to get Badri an EZ-Link card from a vending machine. It was 10 Singapore dollars (₹630) - 5 for the card, and 5 for the balance. I had planned to use my Visa credit card to pay for my own fare. I was relieved to see that my card worked, and I passed through the AFC gates.

We had booked our stay at a hostel named Campbell’s Inn, which was the cheapest we could find in Singapore. It was ₹1500 per night for dorm beds. The hostel was located in Little India. While Little India has an eponymous metro station, the one closest to our hostel was Rochor.

On the way to the hostel, we found out that our booking had been canceled.

We had booked from the Hostelworld website, opting to pay the deposit in advance and to pay the balance amount in person upon reaching. However, Hostelworld still tried to charge Badri’s card again before our arrival. When the unauthorized charge failed, they sent an automatic message saying “we tried to charge” and to contact them soon to avoid cancellation, which we couldn’t do as we were in the plane.

Despite this, we went to the hostel to check the status of our booking.

The trip from the airport to Rochor required a couple of transfers. It was 2 Singapore dollars (approx. ₹130) and took approximately an hour.

Upon reaching the hostel, we were informed that our booking had indeed been canceled, and were not given any reason for the cancelation. Furthermore, no beds were available at the hostel for us to book on the spot.

We decided to roam around and look for accommodation at other hostels in the area. Soon, we found a hostel by the name of Snooze Inn, which had two beds available. It was 36 Singapore dollars per person (around ₹2300) for a dormitory bed. Snooze Inn advertised supporting RuPay cards and UPI. Some other places in that area did the same. We paid using my card. We checked in and slept for a couple of hours after taking a shower.

By the time we woke up, it was dark. We met Praveen’s friend Sabeel to get my FLX1 phone. We also went to Mustafa Center nearby to exchange Indian rupees for Singapore dollars. Mustafa Center also had a shopping center with shops selling electronic items and souvenirs, among other things. When we were dropping off Sabeel at a bus stop, we discovered that the bus stops in Singapore had a digital board mentioning the bus routes for the stop and the number of minutes each bus was going to take.

In addition to an organized bus system, Singapore had good pedestrian infrastructure. There were traffic lights and zebra crossings for pedestrians to cross the roads. Unlike in Indian cities, rules were being followed. Cars would stop for pedestrians at unmanaged zebra crossings; pedestrians would in turn wait for their crossing signal to turn green before attempting to walk across. Therefore, walking in Singapore was easy.

Traffic rules were taken so seriously in Singapore I (as a pedestrian) was afraid of unintentionally breaking them, which could get me in trouble, as breaking rules is dealt with heavy fines in the country. For example, crossing roads without using a marked crossing (while being within 50 meters of it) - also known as jaywalking - is an offence in Singapore.

Moreover, the streets were litter-free, and cleanliness seemed like an obsession.

After exploring Mustafa Center, we went to a nearby 7-Eleven to top up Badri’s EZ-Link card. He gave 20 Singapore dollars for the recharge, which credited the card by 19.40 Singapore dollars (0.6 dollars being the recharge fee).

When I was planning this trip, I discovered that the World Chess Championship match was being held in Singapore. I seized the opportunity and bought a ticket in advance. The next day - the 5th of December - I went to watch the 9th game between Gukesh Dommaraju of India and Ding Liren of China. The venue was a hotel on Sentosa Island, and the ticket was 70 Singapore dollars, which was around ₹4000 at the time.

We checked out from our hostel in the morning, as we were planning to stay with Badri’s aunt that night. We had breakfast at a place in Little India. Then we took a couple of buses, followed by a walk to Sentosa Island. Paying the fare for the buses was similar to the metro - I tapped my credit card in the bus, while Badri tapped his EZ-Link card. We also had to tap it while getting off.

If you are tapping your credit card to use public transport in Singapore, keep in mind that the total amount of all the trips taken on a day is deducted at the end. This makes it hard to determine the cost of individual trips. For example, I could take a bus and get off after tapping my card, but I would have no way to determine how much this journey cost.

When you tap in, the maximum fare amount gets deducted. When you tap out, the balance amount gets refunded (if it’s a shorter journey than the maximum fare one). So, there is incentive for passengers not to get off without tapping out. Going by your card statement, it looks like all that happens virtually, and only one statement comes in at the end. Maybe this combining only happens for international cards.

We got off the bus a kilometer away from Sentosa Island and walked the rest of the way. We went on the Sentosa Boardwalk, which is itself a tourist attraction. I was using Organic Maps to navigate to the hotel Resorts World Sentosa, but Organic Maps’ route led us through an amusement park. I tried asking the locals (people working in shops) for directions, but it was a Chinese-speaking region, and they didn’t understand English. Fortunately, we managed to find a local who helped us with the directions.

Following the directions, we somehow ended up having to walk on a road which did not have pedestrian paths. Singapore is a country with strict laws, so we did not want to walk on that road. Avoiding that road led us to the Michael Hotel. There was a person standing at the entrance, and I asked him for directions to Resorts World Sentosa. The person told me that the bus (which was standing at the entrance) would drop me there! The bus was a free service for getting to Resorts World Sentosa. Here I parted ways with Badri, who went to his aunt’s place.

I got to the Resorts Sentosa and showed my ticket to get in. There were two zones inside - the first was a room with a glass wall separating the audience and the players. This was the room to watch the game physically, and resembled a zoo or an aquarium. :) The room was also a silent room, which means talking or making noise was prohibited. Audiences were only allowed to have mobile phones for the first 30 minutes of the game - since I arrived late, I could not bring my phone inside that room.

The other zone was outside this room. It had a big TV on which the game was being broadcast along with commentary by David Howell and Jovanka Houska - the official FIDE commentators for the event. If you don’t already know, FIDE is the authoritative international chess body.

I spent most of the time outside that silent room, giving me an opportunity to socialize. A lot of people were from Singapore. I saw there were many Indians there as well. Moreover, I had a good time with Vasudevan, a journalist from Tamil Nadu who was covering the match. He also asked questions to Gukesh during the post-match conference. His questions were in Tamil to lift Gukesh’s spirits, as Gukesh is a Tamil speaker.

Tea and coffee were free for the audience. I also bought a T-shirt from their stall as a souvenir.

After the game, I took a shuttle bus from Resorts World Sentosa to a metro station, then travelled to Pasir Ris by metro, where Badri was staying with his aunt. I thought of getting something to eat, but could not find any cafés or restaurants while I was walking from the Pasir Ris metro station to my destination, and was positively starving when I got there.

Badri’s aunt’s place was an apartment in a gated community. On the gate was a security guard who asked me the address of the apartment. Upon entering, there were many buildings. To enter the building, you need to dial the number of the apartment you want to go to and speak to them. I had seen that in the TV show Seinfeld, where Jerry’s friends used to dial Jerry to get into his building.

I was afraid they might not have anything to eat because I told them I was planning to get something on the way. This was fortunately not the case, and I was relieved to not have to sleep with an empty stomach.

Badri’s uncle gave us an idea of how safe Singapore is. He said that even if you forget your laptop in a public space, you can go back the next day to find it right there in the same spot. I also learned that owning cars was discouraged in Singapore - the government imposes a high registration fee on them, while also making public transport easy to use and affordable. I also found out that 7-Eleven was not that popular among residents in Singapore, unlike in Malaysia or Thailand.

The next day was our third and final day in Singapore. We had a bus in the evening to Johor Bahru in Malaysia. We got up early, had breakfast, and checked out from Badri’s aunt’s home. A store by the name of Cat Socrates was our first stop for the day, as Badri wanted to buy some stationery. The plan was to take the metro, followed by the bus. So we got to Pasir Ris metro station. Next to the metro station was a mall. In the mall, Badri found an ATM where our cards worked, and we got some Singapore dollars.

It was noon when we reached the stationery shop mentioned above. We had to walk a kilometer from the place where the bus dropped us. It was a hot, sunny day in Singapore, so walking was not comfortable. We had to go through residential areas in Singapore. We saw some non-touristy parts of Singapore.

After we were done with the stationery shop, we went to a hawker center to get lunch. Hawker centers are unique to Singapore. They have a lot of shops that sell local food at cheap prices. It is similar to a food court. However, unlike the food courts in malls, hawker centers are open-air and can get quite hot.

To have something, you just need to buy it from one of the shops and find a table. After you are done, you need to put your tray in the tray-collecting spots. I had a kaya toast with chai, since there weren’t many vegetarian options. I also bought a persimmon from a nearby fruit vendor. On the other hand, Badri sampled some local non-vegetarian dishes.

Next, we took a metro to Raffles Place, as we wanted to visit Merlion, the icon of Singapore. It is a statue having the head of a lion and the body of a fish. While getting through the AFC gates, my card was declined. Therefore, I had to buy an EZ-Link card, which I had been avoiding because the card itself costs 5 Singapore dollars.

From the Raffles Place metro station, we walked to Merlion. The place also gave a nice view of Marina Bay Sands. It was filled with tourists clicking pictures, and we also did the same.

After this, we went to the bus stop to catch our bus to the border city of Johor Bahru, Malaysia. The bus was more than an hour late, and we worried that we had missed the bus. I asked an Indian woman at the stop who also planned to take the same bus, and she told us that the bus was late. Finally, our bus arrived, and we set off for Johor Bahru.

Before I finish, let me give you an idea of my expenditure. Singapore is an expensive country, and I realized that expenses could go up pretty quickly. Overall, my stay in Singapore for 3 days and 2 nights was approx. 5500 rupees. That too, when we stayed one night at Badri’s aunt’s place (so we didn’t have to pay for accomodation for one of the nights) and didn’t have to pay for a couple of meals. This amount doesn’t include the ticket for the chess game, but includes the costs of getting there. If you are in Singapore, it is likely you will pay a visit to Sentosa Island anyway.

Stay tuned for our experiences in Malaysia!

Credits: Thanks to Dione, Sahil, Badri and Contrapunctus for reviewing the draft. Thanks to Bhe for spotting a duplicate sentence.

]]> Ravi Dwivedi https://www.prashanthudupa.com/emptiness/ https://www.prashanthudupa.com/emptiness/ Sun, 14 Sep 2025 17:02:09 GMT Prashanth Udupa Insight Philosophy https://ravidwivedi.in/posts/installing-debian-with-btrfs-and-encryption/ https://ravidwivedi.in/posts/installing-debian-with-btrfs-and-encryption/ Fri, 29 Aug 2025 20:23:11 GMT /dev/mapper/sda2_crypt / btrfs noatime,compress=zstd,ssd,discard=async,space_cache=v2,subvol=@ 0 0 /dev/mapper/sda2_crypt /home btrfs noatime,compress=zstd,ssd,discard=async,space_cache=v2,subvol=@home 0 0 # /boot was on /dev/sda1 during installation UUID=12842b16-d3b3-44b4-878a-beb1e6362fbc /boot ext4 defaults 0 2 /dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 Please double check the fstab file before saving it. In Nano, you can press Ctrl+O followed by pressing Enter to save the file. Then press Ctrl+X to quit Nano. Now, preview the fstab file by running cat /target/etc/fstab and verify that the entries are correct, otherwise you will booted to an unusable and broken system after the installation is complete. Next, press Ctrl + Alt + F1 to go back to the installer. Proceed to “Install the base system.” Screenshot of Debian installer installing the base system. I chose the default option here - linux-image-amd64. After this, the installer will ask you a few more questions. For desktop environment, I chose KDE Plasma. You can choose the desktop environment as per your liking. I will not cover the rest of the installation process and assume that you were able to install from here. Post installation Let’s jump to our freshly installed Debian system. Since I created a root user, I added the user ravi to the suoders file (/etc/sudoers) so that ravi can run commands with sudo. Follow this if you would like to do the same. Now we set up zram as swap. First, install zram-tools. sudo apt install zram-tools Now edit the file /etc/default/zramswap and make sure to have the following lines are uncommented: ALGO=lz4 PERCENT=50 Now, run sudo systemctl restart zramswap If you run lsblk now, you should see the below-mentioned entry in the output: zram0 253:0 0 7.8G 0 disk [SWAP] This shows us that zram has been activated as swap. Now we install timeshift, which can be done by running sudo apt install timeshift After the installation is complete, run Timeshift and schedule snapshots as you please. We are done now. Hope the tutorial was helpful. See you in the next post and let me know if you have any suggestions and questions on this tutorial.]]> Motivation

On the 8th of August 2025 (a day before the Debian Trixie release), I was upgrading my personal laptop from Debian Bookworm to Trixie. It was a major update. However, the update didn’t go smoothly, and I ran into some errors. From the Debian support IRC channel, I got to know that it would be best if I removed the texlive packages.

However, it was not so easy to just remove texlive with a simple apt remove command. I had to remove the texlive packages from /usr/bin. Then I ran into other errors. Hours after I started the upgrade, I realized I preferred having my system as it was before, as I had to travel to Noida the next day. Needless to say, I wanted to go to sleep rather than fix my broken system. Only if I had a way to go back to my system before I started upgrading, it would have saved a lot of trouble for me. I ended up installing Trixie from scratch.

It turns out that there was a way to recover to the state before the upgrade - using Timeshift to roll back the system to a state (in our example, it is the state before the upgrade process started) in the past. However, it needs the Btrfs filesystem with appropriate subvolumes, not provided by Debian installer in their guided partitioning menu.

I have set it up after a few weeks of the above-mentioned incident. Let me demonstrate how it works.

Check the screenshot above. It shows a list of snapshots made by Timeshift. Some of them were made by me manually. Others were made by Timeshift automatically as per the routine - I have set up hourly backups and weekly backups etc.

In the above-mentioned major update, I could have just taken a snapshot using Timeshift before performing the upgrade and could have rolled back to that snapshot when I found that I cannot spend more time on fixing my installation errors. Then I could just perform the upgrade later.

Installation

In this tutorial, I will cover how I installed Debian with Btrfs and disk encryption, along with creating subvolumes @ for root and @home for /home so that I can use Timeshift to create snapshots. These snapshots are kept on the same disk where Debian is installed, and the use-case is to roll back to a working system in case I mess up something or to recover an accidentally deleted file.

I went through countless tutorials on the Internet, but I didn’t find a single tutorial covering both the disk encryption and the above-mentioned subvolumes (on Debian). Debian doesn’t create the desired subvolumes by default, therefore the process requires some manual steps, which beginners may not be comfortable performing. Beginners can try distros such as Fedora and Linux Mint, as their installation includes Btrfs with the required subvolumes.

Furthermore, it is pertinent to note that I used Debian Trixie’s DVD iso on a real laptop (not a virtual machine) for my installation. Debian Trixie is the codename for the current stable version of Debian. Then I took screenshots in a virtual machine by repeating the process. Moreover, a couple of screenshots are from the installation I did on the real laptop.

Let’s start the tutorial by booting up the Debian installer.

The above screenshot shows the first screen we see on the installer. Since we want to choose Expert Install, we select Advanced Options in the screenshot above.

Let’s select the Expert Install option in the above screenshot. It is because we want to create subvolumes after the installer is done with the partition, and only then proceed to installing the base system. “Non-expert” install modes proceed directly to installing the system right after creating partitions without pausing for us to create the subvolumes.

After selecting the Expert Install option, you will get the screen above. I will skip to partitioning from here and leave the intermediate steps such as choosing language, region, connecting to Wi-Fi, etc. For your reference, I did create the root user.

Let’s jump right to the partitioning step. Select the Partition disks option from the menu as shown above.

Choose Manual.

Select your disk where you would like to install Debian.

Select Yes when asked for creating a new partition.

I chose the msdos option as I am not using UEFI. If you are using UEFI, then you need to choose the gpt option. Also, your steps will (slightly) differ from mine if you are using UEFI. In that case, you can watch this video by the YouTube channel EF Linux in which he creates an EFI partition. As he doesn’t cover disk encryption, you can continue reading this post after following the steps corresponding to EFI.

Select the free space option as shown above.

Choose Create a new partition.

I chose the partition size to be 1 GB.

Choose Primary.

Choose Beginning.

Now, I got to this screen.

I changed mount point to /boot and turned on the bootable flag and then selected “Done setting up the partition.”

Now select free space.

Choose the Create a new partition option.

I made the partition size equal to the remaining space on my disk. I do not intend to create a swap partition, so I do not need more space.

Select Primary.

Select the Use as option to change its value.

Select “physical volume for encryption.”

Select Done setting up the partition.

Now select “Configure encrypted volumes.”

Select Yes.

Select Finish.

Selecting Yes will take a lot of time to erase the data. Therefore, I would say if you have hours for this step (in case your SSD is like 1 TB), then I would recommend selecting “Yes.” Otherwise, you could select “No” and compromise on the quality of encryption.

After this, you will be asked to enter a passphrase for disk encryption and confirm it. Please do so. I forgot to take the screenshot for that step.

Now select that encrypted volume as shown in the screenshot above.

Here we will change a couple of options which will be shown in the next screenshot.

In the Use as menu, select “btrfs journaling file system.”

Now, click on the mount point option.

Change it to “/ - the root file system.”

Select Done setting up the partition.

This is a preview of the paritioning after performing the above-mentioned steps.

If everything is okay, proceed with the Finish partitioning and write changes to disk option.

The installer is reminding us to create a swap partition. I proceeded without it as I planned to add swap after the installation.

If everything looks fine, choose “yes” for writing the changes to disks.

Now we are done with partitioning and we are shown the screen in the screenshot above. If we had not selected the Expert Install option, the installer would have proceeded to install the base system without asking us.

However, we want to create subvolumes before proceeding to install the base system. This is the reason we chose Expert Install.

Now press Ctrl + F2.

You will see the screen as in the above screenshot. It says “Please press Enter to activate this console.” So, let’s press Enter.

After pressing Enter, we see the above screen.

The screenshot above shows the steps I performed in the console. I followed the already mentioned video by EF Linux for this part and adapted it to my situation (he doesn’t encrypt the disk in his tutorial).

First we run df -h to have a look at how our disk is partitioned. In my case, the output was:

# df -h
Filesystem              Size  Used  Avail   Use% Mounted on
tmpfs                   1.6G  344.0K  1.6G    0% /run
devtmpfs                7.7G       0  7.7G   0% /dev
/dev/sdb1               3.7G    3.7G    0   100% /cdrom
/dev/mapper/sda2_crypt  952.9G  5.8G  950.9G  0% /target
/dev/sda1               919.7M  260.0K  855.8M  0% /target/boot

df -h shows us that /dev/mapper/sda2_crypt and /dev/sda1 are mounted on /target and /target/boot respectively.

Let’s unmount them. For that, we run:

# umount /target
# umount /target/boot

Next, let’s mount our root filesystem to /mnt.

# mount /dev/mapper/sda2_crypt /mnt

Let’s go into the /mnt directory.

# cd /mnt

Upon listing the contents of this directory, we get:

/mnt # ls
@rootfs

Debian installer has created a subvolume @rootfs automatically. However, we need the subvolumes to be @ and @home. Therefore, let’s rename the @rootfs subvolume to @.

/mnt # mv @rootfs @

Listing the contents of the directory again, we get:

/mnt # ls
@

We only one subvolume right now. Therefore, let us go ahead and create another subvolume @home.

/mnt # btrfs subvolume create @home
Create subvolume './@home'

If we perform ls now, we will see there are two subvolumes:

/mnt # ls
@ @home

Let us mount /dev/mapper/sda2_crypt to /target

/mnt # mount -o noatime,space_cache=v2,compress=zstd,ssd,discard=async,subvol=@ /dev/mapper/sda2_crypt /target/

Now we need to create a directory for /home.

/mnt # mkdir /target/home/

Now we mount the /home directory with subvol=@home option.

/mnt # mount -o noatime,space_cache=v2,compress=zstd,ssd,discard=async,subvol=@home /dev/mapper/sda2_crypt /target/home/

Now mount /dev/sda1 to /target/boot.

/mnt # mount /dev/sda1 /target/boot/

Now we need to add these options to the fstab file, which is located at /target/etc/fstab. Unfortunately, vim is not installed in this console. The only way to edit is Nano.

nano /target/etc/fstab

Edit your fstab file to look similar to the one in the screenshot above. I am pasting the fstab file contents below for easy reference.

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# systemd generates mount units based on this file, see systemd.mount(5).
# Please run 'systemctl daemon-reload' after making changes here.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/mapper/sda2_crypt /        btrfs   noatime,compress=zstd,ssd,discard=async,space_cache=v2,subvol=@ 0       0
/dev/mapper/sda2_crypt /home    btrfs   noatime,compress=zstd,ssd,discard=async,space_cache=v2,subvol=@home 0       0
# /boot was on /dev/sda1 during installation
UUID=12842b16-d3b3-44b4-878a-beb1e6362fbc /boot           ext4    defaults        0       2
/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0

Please double check the fstab file before saving it. In Nano, you can press Ctrl+O followed by pressing Enter to save the file. Then press Ctrl+X to quit Nano. Now, preview the fstab file by running

cat /target/etc/fstab

and verify that the entries are correct, otherwise you will booted to an unusable and broken system after the installation is complete.

Next, press Ctrl + Alt + F1 to go back to the installer.

Proceed to “Install the base system.”

I chose the default option here - linux-image-amd64.

After this, the installer will ask you a few more questions. For desktop environment, I chose KDE Plasma. You can choose the desktop environment as per your liking. I will not cover the rest of the installation process and assume that you were able to install from here.

Post installation

Let’s jump to our freshly installed Debian system. Since I created a root user, I added the user ravi to the suoders file (/etc/sudoers) so that ravi can run commands with sudo. Follow this if you would like to do the same.

Now we set up zram as swap. First, install zram-tools.

sudo apt install zram-tools

Now edit the file /etc/default/zramswap and make sure to have the following lines are uncommented:

ALGO=lz4
PERCENT=50

Now, run

sudo systemctl restart zramswap

If you run lsblk now, you should see the below-mentioned entry in the output:

zram0          253:0    0   7.8G  0 disk  [SWAP]

This shows us that zram has been activated as swap.

Now we install timeshift, which can be done by running

sudo apt install timeshift

After the installation is complete, run Timeshift and schedule snapshots as you please. We are done now. Hope the tutorial was helpful.

See you in the next post and let me know if you have any suggestions and questions on this tutorial.

]]> Ravi Dwivedi https://www.learningfromdata.zingg.ai/p/london-calling-do-you-really-know https://www.learningfromdata.zingg.ai/p/london-calling-do-you-really-know Tue, 26 Aug 2025 18:22:30 GMT

Last week, I was finetuning the Zingg website, when something caught my eye. We have a quote on Fortnum & Mason’s journey from fragmented customer records to a unified view with Zingg:

"For the first time, we're able to understand how customers are shopping with us—online, in-store, over the phone, or in restaurants. We never had that before."

That sentence got me thinking: in an era where we can track a customer's digital breadcrumbs across every touchpoint, why do so many brands still struggle with the simple question: "Do you really know your customer?"

The Identity Resolution Promise That Keeps Getting Broken

Most of us have watched the same promises get made over and over again. First, it was Master Data Management (MDM) systems that would create "one golden record." Then identity services like LiveRamp promised to connect everyone's messy data through their identity graph. Then for the last few years, Customer Data Platforms (CDPs) became our solution to the elusive single customer view.

But here's what I keep seeing: enterprises continue to struggle with messy and fragmented data. Multi-year MDM implementations that are still not very useful. The MDM system could tell you the exact lineage of how John Smith's email address made it through seven data transformation steps, but it couldn't figure out that J. Smith from the mobile app was the same person. The system was simply not ready for the messy reality of how customer data actually behaves.

The same pattern plays out with identity services. A different promise: "Don't worry about cleaning your own house — we'll connect everyone's messy data." Unfortunately, identity services operate as closed systems where you can't see or validate their matching logic. When they say Customer A matches Customer B, you're essentially trusting a proprietary algorithm without transparency. This becomes problematic when you need to adjust matching to your specific business context, explain results to stakeholders and debug why certain matches seem incorrect. Add to that the privacy risk of sending trusted data to a third party. Designed around fixed data models (email, phone, cookie IDs, etc.), it is tough to resolve on more complex or domain-specific attributes (like healthcare patient IDs, loyalty card numbers, or industry-specific identifiers).

CDPs were supposed to fix everything by democratizing customer data for marketers. The reality? Most became expensive data silos with pretty dashboards, lots of data movement yet the hard work of actually resolving who is who got pushed to the background.

What Actually Works

This is why I'm excited about what the team at Fortnum & Mason have built. Starting with data scattered across restaurant bookings, email sign-ups, online orders, and in-store transactions, they built a composable CDP using Zingg for entity resolution on Databricks.

The transformation wasn't about technology for technology's sake.

"We are able to start turning that into insights and use it to personalize experiences online. We are going to start doing this in store and in our restaurants as well. Being able to connect all of these different customer experiences together is really important for us."

What made their approach different? They treated entity resolution as a first-class capability, not an afterthought. They started with Zingg's open source version to build confidence, understood how the matching worked, then moved to the enterprise version. The entire journey from proof of concept to production took just 2-3 months. For the first time, they can understand how customers shop across all channels. Suddenly, “one customer” meant the same thing everywhere — and the results have been transformational, not just for analytics, but for personalization and customer experience.

Or look at the Orthodox Union, a non-profit serving millions globally. They struggled with donor and member records scattered across legacy systems, event platforms, and CRM databases. Without a reliable way to unify identities, their engagement strategies risked missing context and duplicating outreach. By resolving identities natively in their Snowflake data warehouse, they were able to create a consistent single view of donors and members, while respecting privacy and compliance requirements central to their mission. For them, identity resolution wasn’t just a data challenge — it was a way to deepen community trust.

These are two very different organizations, but the pattern is the same: composability delivers agility, while identity delivers coherence. Without both, the vision of modern customer experience falls short.

The Composable CDP Revolution

Fortnum & Mason's and Orthodox Union’s approach reflects a broader trend that's reshaping the market. Composable CDPs sit on top of existing data clouds to preserve the customer 360 and activate real-time data across channel tools. Instead of moving data around, they work where the data lives. They are private by design. They leverage existing investments in governance and observability. They scale as you grow.

But here’s the hard truth: Composable architectures give you flexibility — but only if the components are stitched together with a stable, reliable, and explainable identity layer. You can have the most elegant semantic layer, real-time pipelines, and activation tools, but if “Acme Corp” and “ACME Inc.” are treated as two different customers, your insights fracture. If one table says “Tim Chen” and another says “T. Chen,” your campaigns lose relevance. If customer data is duplicated at ingestion, those cracks echo through every model, dashboard, and recommendation.

And here’s the other piece we don’t talk about enough: owning your identity layer means owning your customer understanding. Instead of shipping your most sensitive data out to a black-box vendor, you resolve identities where your data already lives — in your own warehouse or lake. That means:

• Data ownership: You stay in control of your customer data, critical in a world of tightening privacy regulations.

• Schema flexibility: As your business evolves — new channels, attributes, and data types — your identity system evolves with it, instead of breaking because it was designed for someone else’s schema.

• Privacy and compliance: By resolving identities within your existing data governance framework, you respect customer consent and meet regulatory requirements without creating extra data copies.

• Stack leverage: You’re not rebuilding infrastructure. You’re making your existing investments in warehouses, lakes, and analytics more powerful by adding the missing link: identity.

Why I'm Hosting This Roundtable

This brings me to London. On 9th September, I'll be at the Martech World Forum hosting a roundtable titled "Do You Really Know Your Customer?" — and it will be lovely to meet you if you are there!

Here's what we'll cover:

The Identity Crisis in Marketing Why traditional approaches to customer identity keep failing, and how agentic AI makes the problem more urgent than ever.

Building Composable CDPs That Actually Work Moving beyond tool marketing to understand what composable really means, when it makes sense, and how to implement it successfully.

The Zingg Approach to Entity Resolution How modern ML-based entity resolution differs from traditional MDM, why transparency matters, and how to get started with open source tools.

Real Customer Stories on fragmented data to unified customer understanding, including the practical challenges and business outcomes. The folks from Fortnum & Mason will join us to share their real-world experience, the challenges they overcame, and the business impact they've achieved.

The roundtable format means we can go deep, ask tough questions, and learn from each other's experiences. Whether you're just starting your customer identity journey or you've been through multiple implementations, there is value in understanding what's working now and what's coming next.

Join Us in London

I have a limited number of VIP passes available for the Martech World Forum. If you're a data or marketing leader in London, I'd love to share the passes and have you join the conversation in person. Because at the end of the day, the real question isn’t just “do you know your customer?” — it’s whether your systems know them too.

Not just their demographic data or their transaction history, but their actual journey across all your touchpoints? Can you recognize them when they call customer service after browsing your website? Do you know that the person who signed up for your newsletter is the same one who made a purchase in your store?

If you can't answer these questions confidently, you're not alone. But you also can't afford to stay in that position much longer.

The companies that figure this out — the ones that truly know their customers — are going to have an unfair advantage in the age of AI-driven experiences. They'll deliver more relevant products, more timely services, and more personal interactions. They'll waste less money on duplicate processes and misguided campaigns.

Most importantly, they'll build the kind of customer relationships that survive economic downturns, competitive pressures, and technology transitions.

And if you can't make it to London but have war stories about identity resolution initiatives (successful or otherwise), I'd love to hear them. This isn't really about technology — it's about understanding the humans behind the data. And that, I've learned over the years, is both the hardest and most rewarding part of what we do.

---

Want to learn more about entity resolution and composable CDPs? Check out Zingg's open source project or read about how we're helping companies like Fortnum & Mason build better customer understanding.

---

]]> Sonal Goyal https://www.prashanthudupa.com/beyond-self-improvement-letting-go-of-the-separate-self/ https://www.prashanthudupa.com/beyond-self-improvement-letting-go-of-the-separate-self/ Wed, 20 Aug 2025 14:08:29 GMT Prashanth Udupa Insight Philosophy https://www.learningfromdata.zingg.ai/p/agentic-ai-is-only-as-smart-as-its https://www.learningfromdata.zingg.ai/p/agentic-ai-is-only-as-smart-as-its Wed, 13 Aug 2025 09:43:38 GMT A few weeks ago, I was talking to a data leader at a fast-growing retail brand.
They’d just rolled out their shiny new AI marketing agent. It could segment customers, design campaigns, and send hyper-personalized emails — all without human intervention.

Two weeks in, the VP of Marketing noticed something strange. One of their top VIP customers had received three different “Welcome to our brand!” offers — each with a different discount.

Turns out, their CRM had John A. Smith, J. Smith, and Jonathan Smith as three separate customers.

The agent wasn’t broken. It was doing exactly what it was told — treat each record as a unique person and delight them with a welcome offer. The problem? The records were wrong.

And this, I’ve realized over and over again, is where agentic AI stumbles: when the entities are messy, the automation multiplies the mess.

Over the past few years, I’ve seen this pattern play out in every industry:

Retail — Duplicate customer records mean multiple shipments, multiple offers, multiple refunds. Margins erode quietly.

Finance — Fraudsters open several accounts with tiny name variations. Without linking those identities, your fraud-detection AI treats them as separate people — and misses the pattern.

Healthcare — Patient “Mary Chen” at Hospital A is “M. Y. Chen” at Clinic B. An AI care coordinator misses her allergy record and books a risky procedure.

In each case, the AI wasn’t “dumb.” It was confidently acting on bad data. And because agentic AI is autonomous, it didn’t just make one mistake — it repeated the mistake hundreds or thousands of times before anyone noticed. Smart agents get the “who” wrong and this really hurts the business.

If you capture the hype, everyone talks about the cost of building and running agentic AI. Yet, fewer people talk about the cost of running it on messy data. Agents run on LLM calls, and every action burns tokens. When your entities aren’t resolved, the agent:

• Processes duplicates

• Repeats actions unnecessarily

• Sends extra API calls trying to reconcile

Here’s the math from just one example scenario:

*Based on $15 per 1M tokens (GPT-4.1 tier).

Even with cheaper models, the % waste stays the same. That’s just compute waste. It doesn’t even include the cost of fixing the damage — reissuing refunds, restoring trust, or dealing with regulatory fallouts.

---

Why I care about this (and what we built at Zingg)

Before I started Zingg, I kept running into the same frustration:
We had amazing ML models, sophisticated rules engines, smart agents… but they were all reasoning on fragmented, messy entities.

At Zingg, we decided to treat entity resolution as a first-class product, not a one-off data cleanup exercise. That means:

• Matching at scale across CRMs, EHRs, billing systems, and data lakes

• Adaptive learning so the matching improves with every variation we see

• Transparent decisions so you can explain why two records were linked

When customers put this layer under their agentic AI, they stopped paying for their agents to repeatedly do the wrong thing. In one case, a bank’s fraud detection improved 4x — without touching the AI model.

---

My advice to data leaders

If you’re about to launch an agentic AI project, take a breath.
Before you buy another GPU hour or fine-tune another model, ask:

“Do my agents actually know who they’re dealing with?”

Because in my experience, agentic AI is the race car. Entity resolution is the track.
You wouldn’t run a Formula 1 machine on gravel and expect to win.

---

P.S. If you’re curious about what we are building at Zingg, or have war stories about entity chaos in AI systems, I’d love to hear from you.

---

]]> Sonal Goyal https://www.prashanthudupa.com/the-matrix-got-it-wrong/ https://www.prashanthudupa.com/the-matrix-got-it-wrong/ Fri, 08 Aug 2025 15:52:53 GMT Prashanth Udupa Insight Philosophy Uncategorized https://www.evalapply.org/posts/lessons-from-reimplementing-clojure-comp-function/index.html https://www.evalapply.org/posts/lessons-from-reimplementing-clojure-comp-function/index.html Fri, 08 Aug 2025 00:00:00 GMT Aditya Athalye clojure functional_programming howto riff https://ravidwivedi.in/posts/vietnam-visa/ https://ravidwivedi.in/posts/vietnam-visa/ Tue, 05 Aug 2025 22:55:05 GMT In December 2024, Badri and I went to Vietnam. In this post, I’ll document our experiences with the visa process of Vietnam. Vietnam requires an e-visa to enter the country. The official online portal for the e-visa application is evisa.xuatnhapcanh.gov.vn/. However, I submitted my visa application on the website vietnamvisa.govt.vn. It was only after submitting my application and making the payment that I realized that it’s not the official e-visa website. The realization came from the tagline mentioned in the top left corner of the website - the best way to obtain a Vietnam visa.

I was a bit upset that I got tricked by that website. I should have checked the top level domains of Vietnam’s government websites. Anyways, it is pretty easy to confuse govt.vn with gov.vn. I also paid double the amount of the official visa fee. However, I wasn’t asked to provide a flight reservation or hotel bookings - documents which are usually asked for most of the visas. But they did ask me for a photo. I was not even sure whether the website was legit or not.

Badri learnt from my experience and applied through the official Vietnam government website. During the process, he had to provide a hotel booking as well as enter the hotel address into the submission form. Additionally, the official website asked to provide the exact points of entry to and exit from the country, which the non-official website did not ask for. On the other hand, he had to pay only 25 USD versus my 54 USD.

It turned out that the website I registered on was also legit, as they informed me a week later that my visa has been approved, along with a copy of my visa. Further, I was not barred from entering and found to be holding a fake visa. It appears that the main “scam” is not about the visa being fake, but rather that you will be charged more than if you apply through the official website.

I would still recommend you (the readers) to submit your visa application only through the official website and not on any of the other such websites.

Our visa was valid for a month (my visa was valid from the 4th of December 2024 to the 4th of January 2025). We also had a nice time in Vietnam. Stay tuned for my Vietnam travel posts!

Credits to Badri for proofreading and writing his part of the experience.

]]> Ravi Dwivedi https://www.prashanthudupa.com/turiya-pure-consciousness/ https://www.prashanthudupa.com/turiya-pure-consciousness/ Sun, 03 Aug 2025 09:27:49 GMT Prashanth Udupa Philosophy https://www.prashanthudupa.com/pull-back-from-the-machine/ https://www.prashanthudupa.com/pull-back-from-the-machine/ Sat, 02 Aug 2025 06:30:48 GMT Prashanth Udupa Insight Philosophy https://www.prashanthudupa.com/observer-observed/ https://www.prashanthudupa.com/observer-observed/ Fri, 01 Aug 2025 07:55:41 GMT Prashanth Udupa Insight Moments Philosophy https://mrkaran.dev/posts/wireguard-route-fix/ https://mrkaran.dev/posts/wireguard-route-fix/ Wed, 30 Jul 2025 00:00:00 GMT 10.100.0.2.7778: Flags [S], seq 324784341, win 42780, ... The TCP SYN packet arrived successfully through wg0. But no response. No SYN-ACK (success), no ICMP error (rejection). The packet was being silently dropped. The Culprit: firewalld# The client was running Arch Linux with firewalld. My mistake was trying to manage firewall rules with iptables commands in the WireGuard PostUp script. While iptables was installed, firewalld was the active manager, using nftables as its backend. When a new interface like wg0 comes up, firewalld needs to know which “zone” it belongs to. If unassigned, it gets handled by a restrictive default policy that silently DROPs unsolicited TCP packets while allowing ICMP (pings). The Fix# Don’t add iptables rules. Just assign the WireGuard interface to the right firewalld zone. For internal tunnels, trusted works well. On the client: sudo firewall-cmd --permanent --zone=trusted --add-interface=wg0 sudo firewall-cmd --reload TCP connections worked instantly after this. TL;DR: If WireGuard pings work but TCP fails with “no route to host”, it’s probably a client firewall issue. On firewalld systems, assign the WireGuard interface to the right zone instead of messing with iptables. Fin!]]> I recently spent some time debugging a WireGuard tunnel that was acting weird. The handshake was successful, pings worked perfectly, but any TCP connection failed with connect: no route to host.

Classic misleading error message. The routing was fine.

The Setup#

Server with a public IP running WireGuard (wg0) with IP 10.100.0.1/24. Client connects and gets assigned 10.100.0.2/32. I wanted to proxy TCP traffic from the server to a service running on the client at 10.100.0.2:7778.

The Investigation#

Diagnostics showed contradictory results:

Routing worked fine: Server routing table correctly directed 10.100.0.0/24 traffic to wg0. Pings were successful:

# On the server
$ ping -c 3 10.100.0.2
PING 10.100.0.2 (10.100.0.2) 56(84) bytes of data.
64 bytes from 10.100.0.2: icmp_seq=1 ttl=64 time=150 ms
...

TCP failed immediately:

# On the server
$ curl -v http://10.100.0.2:7778
* Trying 10.100.0.2:7778...
* connect to 10.100.0.2 port 7778 from 10.100.0.1 port 59812 failed: No route to host

The key insight: ICMP was being treated differently than TCP. This pointed to a firewall issue, not routing. The “no route to host” error was the kernel interpreting an ICMP “Destination Unreachable” message from the remote peer.

But when I ran tcpdump on the client, things got stranger:

# On the client
$ sudo tcpdump -i any -n 'host 10.100.0.1'

# Output when the server tries to connect
17:36:03.043147 wg0 In  IP 10.100.0.1.14808 > 10.100.0.2.7778: Flags [S], seq 324784341, win 42780, ...

The TCP SYN packet arrived successfully through wg0. But no response. No SYN-ACK (success), no ICMP error (rejection). The packet was being silently dropped.

The Culprit: firewalld#

The client was running Arch Linux with firewalld. My mistake was trying to manage firewall rules with iptables commands in the WireGuard PostUp script. While iptables was installed, firewalld was the active manager, using nftables as its backend.

When a new interface like wg0 comes up, firewalld needs to know which “zone” it belongs to. If unassigned, it gets handled by a restrictive default policy that silently DROPs unsolicited TCP packets while allowing ICMP (pings).

The Fix#

Don’t add iptables rules. Just assign the WireGuard interface to the right firewalld zone. For internal tunnels, trusted works well.

On the client:

sudo firewall-cmd --permanent --zone=trusted --add-interface=wg0
sudo firewall-cmd --reload

TCP connections worked instantly after this.

TL;DR: If WireGuard pings work but TCP fails with “no route to host”, it’s probably a client firewall issue. On firewalld systems, assign the WireGuard interface to the right zone instead of messing with iptables.

Fin!

]]> Karan Sharma https://ravidwivedi.in/posts/how-to-paste-password-on-bank-site/ https://ravidwivedi.in/posts/how-to-paste-password-on-bank-site/ Tue, 29 Jul 2025 07:38:51 GMT If your bank is like mine, its website doesn’t allow you to copy your password and paste it by performing a simple Ctrl+V. I tried the Don’t Fuck With Paste extension in Firefox, which could paste my bank account’s profile password but not the login password.

Therefore, I asked on Mastodon a couple of days ago and got some responses. The solution that worked for me was to use Shift+Insert to paste the password. It worked for me in LibreWolf and Firefox, and that’s all I needed.

Furthermore, this behavior by bank websites leads to users choosing insecure and memorable passwords. Using this trick will help you choose strong passwords for your bank account.

I prefer to use random and strong passwords generated using the password manager pass. It is a freedom-respecting software, unlike popular proprietary password managers promoted by YouTubers. Feel free to check out their webpage here. The reason I use pass is that it stores all the passwords locally (and optionally in a remote Git repository) in encrypted form, which can only be decrypted using your private GPG keys.

]]> Ravi Dwivedi https://www.prashanthudupa.com/morality-choice-free-will-and-responsibility/ https://www.prashanthudupa.com/morality-choice-free-will-and-responsibility/ Fri, 25 Jul 2025 06:29:16 GMT Prashanth Udupa Insight Philosophy https://www.prashanthudupa.com/on-memory/ https://www.prashanthudupa.com/on-memory/ Sun, 20 Jul 2025 08:18:16 GMT Prashanth Udupa Insight Philosophy https://www.prashanthudupa.com/maya/ https://www.prashanthudupa.com/maya/ Tue, 15 Jul 2025 17:06:50 GMT Prashanth Udupa Insight Philosophy https://www.evalapply.org/posts/poor-mans-time-oriented-data-system/index.html https://www.evalapply.org/posts/poor-mans-time-oriented-data-system/index.html Mon, 14 Jul 2025 00:00:00 GMT Aditya Athalye functional_programming clojure databases architecture software_design local_first programming web_development https://www.prashanthudupa.com/there-is-no-doer/ https://www.prashanthudupa.com/there-is-no-doer/ Tue, 08 Jul 2025 05:06:22 GMT Prashanth Udupa Insight Philosophy https://www.evalapply.org/posts/llms-are-diamonds/index.html https://www.evalapply.org/posts/llms-are-diamonds/index.html Tue, 01 Jul 2025 00:00:00 GMT Aditya Athalye meta riff ai intelligence_augmentation tools_for_thought https://ravidwivedi.in/posts/brunei-visa/ https://ravidwivedi.in/posts/brunei-visa/ Sat, 21 Jun 2025 08:00:51 GMT In December 2024, my friend Badri and I were planning a trip to Southeast Asia. At this point, we were planning to visit Singapore, Malaysia and Vietnam. My Singapore visa had already been approved, and Malaysia was visa-free for us. For Vietnam, we had to apply for an e-visa online.

We considered adding Brunei to our itinerary. I saw some videos of the Brunei visa process and got the impression that we needed to go to the Brunei embassy in Kuching, Malaysia in person.

However, when I happened to search for Brunei on Organic Maps¹, I stumbled upon the Brunei Embassy in Delhi. It seemed to be somewhere in Hauz Khas. As I was going to Delhi to collect my Singapore visa the next day, I figured I’d also visit the Brunei Embassy to get information about the visa process.

The next day I went to the location displayed by Organic Maps. It was next to the embassy of Madagascar, and a sign on the road divider confirmed that I was at the right place.

That said, it actually looked like someone’s apartment. I entered and asked for directions to the Brunei embassy, but the people inside did not seem to understand my query. After some back and forth, I realized that the embassy wasn’t there.

I now searched for the Brunei embassy on the Internet, and this time I got an address in Vasant Vihar. It seemed like the embassy had been moved from Hauz Khas to Vasant Vihar. Going by the timings mentioned on the web page, the embassy was closing in an hour.

I took a Metro from Hauz Khas to Vasant Vihar. After deboarding at the Vasant Vihar metro station, I took an auto to reach the embassy. The address listed on the webpage got me into the correct block. However, the embassy was still nowhere to be seen. I asked around, but security guards in that area pointed me to the Burundi embassy instead.

After some more looking around, I did end up finding the embassy. I spoke to the security guards at the gate and told them that I would like to know the visa process. They dialled a number and asked that person to tell me the visa process.

I spoke to a lady on the phone. She listed the documents required for the visa process and mentioned that the timings for visa application were from 9 o’clock to 11 o’clock in the morning. She also informed me that the visa fees was ₹1000.

I also asked about the process Badri, who lives far away in Tamil Nadu and cannot report at the embassy physically. She told me that I can submit a visa application on his behalf, along with an authorization letter.

Having found the embassy in Delhi was a huge relief. The other plan - going to Kuching, Malaysia - was a bit uncertain, and we didn’t know how much time it would take. Getting our passport submitted at an embassy in a foreign country was also not ideal.

A few days later, Badri sent me all the documents required for his visa. I went to the embassy and submitted both the applications. The lady who collected our visa submissions asked me for our flight reservations from Delhi to Brunei, whereas ours were (keeping with our itinerary) from Kuala Lampur. She said that she might contact me later if it was required.

For reference, here is the list of documents we submitted -

• Visa application form
• Passport
• A photocopy of passport
• Authorization letter from Badri (authorizing me to submit his application on his behalf)
• Airline ticket itinerary
• Hotel bookings
• Cover letter
• 2 photos
• Proof of employment
• 6 months bank statement (they specifically asked for ₹1,00,000 or more in bank balance)

I then asked about the procedure to collect the passports and visa results. Usually, embassies will tell you that they will contact you when they have decided on your applications. However, here I was informed that if they don’t contact me within 5 days, I can come and collect our passports and visa result between 13:30-14:30 hours on the fifth day. That was strange :)

I did visit the embassy to collect our visa results on the fifth day. However, the lady scolded me for not bringing the receipt she gave me. I was afraid that I might have to go all the way back home and bring the receipt to get our passports. The travel date was close, and it would take some time for Badri to receive his passport via courier as well.

Fortunately, she gave me our passports (with the visa attached) and asked me to share a scanned copy of the receipt via email after I get home.

We were elated that our visas were approved. Now we could focus on booking our flights.

If you are going to Brunei, remember to fill their arrival card from the website within 48 hours of your arrival!

Thanks to Badri and Contrapunctus for reviewing the draft before publishing the article.

---

1. Nowadays, I prefer using Comaps instead of Organic Maps and recommend you do the same. Organic Maps had some issues with its governance and the community issues weren’t being addressed. ↩︎

]]> Ravi Dwivedi https://kcsrk.info/ocaml/x-ocaml/blogging/2025/06/20/xocaml/ https://kcsrk.info/ocaml/x-ocaml/blogging/2025/06/20/xocaml/ Fri, 20 Jun 2025 10:00:00 GMT tag. The snippet below: print_endline "Hello, world" renders to: The code is interpreted in the browser thanks to the OCaml interpreter compiled to JavaScript through the Js_of_ocaml compiler. There is also support for Merlin and OCamlformat in the code editor. Try hovering over the functions and writing some code. You should see inferred types and auto-completion suggestions. It turns out that this solution integrates well with Jekyll, which is what I use for this blog. Reverse-mode AD using Effects Js_of_ocaml also supports effect handlers. Here’s the implementation of reverse-mode algorithmic differentiation, implemented using effect handlers running in the browser. Here are some tests. Using other libraries x-ocaml also supports loading any js_of_ocaml compatible library into the webpage. Let’s use digestif. For any library that you want to export, install the library using opam. x-ocaml provide a command-line utility to export the library. $ x-ocaml --effects digestif.ocaml -o digestif.js This produces the JavaScript artifact that can be used in the webpage. It may be instructive to look at the source of this post to see how the compiler and the libraries are integrated into this blog post. There is a little script at the top of the file: What next? There is a number of rough edges to x-ocaml. This is expected since this project appears to be one of Arthur’s hacking expeditions (which, as usual, is pushing the state of the art forward). It would be fun to use this for teaching CS3100 and also other OCaml tutorials. Perhaps even have an interactive version of Real World OCaml book. Not all OCaml libraries can be compiled to JavaScript. The common reason being that they may depend on features not available on JavaScript. In writing this post, I unsuccessfully tried for a long time to get mirage-cypto working. mirage-crypto has a large C dependency, which does not work with Js_of_ocaml. Js_of_ocaml promises to take any opam library installed on your opam switch and compiles that to JavaScript. However, at that point, we’re really cross compiling the opam packages installed on your switch to JavaScript since the installed package may make some assumptions about the platform that it is supposed to run on. Hence, JavaScript compilation of arbitrary OCaml packages is unlikely to work in the general case. Unfortunately, the error was difficult to debug since the failure was at runtime, and was not apparent in the error messages (at least for me, who has little JavaScript experience). It would be nice to have the opam packages explicitly say whether they are JavaScript compatible, and have build tooling that reports errors like these early.]]> Can we have OCaml notebooks as pure client-side code? Can these notebooks have rich editor support (highlighting, formatting, types on hover, autocompletion, inline diagnostics, etc.)? Can you take packages from OPAM and use them in these notebooks?

The answer to all of these turns out to be a resounding yes thanks for x-ocaml. This post is my experiment playing with x-ocaml and integrating that into this blog.

The most wonderful thing about programming is that it lets you experiment freely. You can try out an idea, get instant feedback, and learn by doing—much like playing with Lego bricks or sketching on a canvas. The two main courses that I teach at IITM, CS3100 and CS6225, both involve me live-coding during every lecture. However, blogging about OCaml where the code is static and non-interactive always felt a bit unsatisfying.

Enter x-ocaml, which allows for a way to embed OCaml notebooks into any webpage thanks to WebComponents. All you need to do is to load some JavaScript in your webpage and you can start embedding code cells using <x-ocaml> tag. The snippet below:

<x-ocaml>
print_endline "Hello, world"
</x-ocaml>

renders to:

print_endline "Hello, world"

The code is interpreted in the browser thanks to the OCaml interpreter compiled to JavaScript through the Js_of_ocaml compiler. There is also support for Merlin and OCamlformat in the code editor. Try hovering over the functions and writing some code. You should see inferred types and auto-completion suggestions. It turns out that this solution integrates well with Jekyll, which is what I use for this blog.

Reverse-mode AD using Effects

Js_of_ocaml also supports effect handlers. Here’s the implementation of reverse-mode algorithmic differentiation, implemented using effect handlers running in the browser.

open Effect open Effect.Deep module F : sig type t val mk : float -> t val ( +. ) : t -> t -> t val ( *. ) : t -> t -> t val grad : (t -> t) -> float -> float val grad2 : (t * t -> t) -> float * float -> float * float end = struct type t = { v : float; mutable d : float } let mk v = { v; d = 0.0 } type _ eff += Add : t * t -> t eff type _ eff += Mult : t * t -> t eff let run f = ignore (match f () with | r -> r.d <- 1.0; r; | effect (Add(a,b)), k -> let x = {v = a.v +. b.v; d = 0.0} in ignore (continue k x); a.d <- a.d +. x.d; b.d <- b.d +. x.d; x | effect (Mult(a,b)), k -> let x = {v = a.v *. b.v; d = 0.0} in ignore (continue k x); a.d <- a.d +. (b.v *. x.d); b.d <- b.d +. (a.v *. x.d); x) let grad f x = let x = mk x in run (fun () -> f x); x.d let grad2 f (x, y) = let x, y = (mk x, mk y) in run (fun () -> f (x, y)); (x.d, y.d) let ( +. ) a b = perform (Add (a, b)) let ( *. ) a b = perform (Mult (a, b)) end

Here are some tests.

(* XXX KC: `assert` from standard library doesn't work. Why? *) let assert' c = if not c then raise (Failure "assertion failed!") let test1 = (* f = x + x^3 => df/dx = 1 + 3 * x^2 *) for x = 0 to 10 do let x = float_of_int x in let d1 = F.(grad (fun x -> x +. (x *. x *. x)) x) in let d2 = 1.0 +. (3.0 *. x *. x) in Printf.printf "%f %f\n" d1 d2; assert' ( d1 = d2 ) done let test2 = (* f = x^2 + x^3 => df/dx = 2*x + 3 * x^2 *) for x = 0 to 10 do let x = float_of_int x in assert' ( F.(grad (fun x -> (x *. x) +. (x *. x *. x)) x) = (2.0 *. x) +. (3.0 *. x *. x)) done let test3 = (* f = x^2 * y^4 => df/dx = 2 * x * y^4 df/dy = 4 * x^2 * y^3 *) for x = 0 to 10 do for y = 0 to 10 do let x = float_of_int x in let y = float_of_int y in assert' ( F.(grad2 (fun (x, y) -> x *. x *. y *. y *. y *. y) (x, y)) = (2.0 *. x *. y *. y *. y *. y, 4.0 *. x *. x *. y *. y *. y)) done done

Using other libraries

x-ocaml also supports loading any js_of_ocaml compatible library into the webpage. Let’s use digestif.

For any library that you want to export, install the library using opam. x-ocaml provide a command-line utility to export the library.

$ x-ocaml --effects digestif.ocaml -o digestif.js

This produces the JavaScript artifact that can be used in the webpage. It may be instructive to look at the source of this post to see how the compiler and the libraries are integrated into this blog post. There is a little script at the top of the file:

<script async
  src="{{ '/assets/x-ocaml.js' | absolute_url }}"
  src-worker="{{ '/assets/x-ocaml.worker+effects.js' | absolute_url }}"
  src-load="{{ '/assets/digestif.js' | absolute_url }}"
></script>

let hash = Digestif.MD5.(digest_string "hello" |> to_hex)

What next?

There is a number of rough edges to x-ocaml. This is expected since this project appears to be one of Arthur’s hacking expeditions (which, as usual, is pushing the state of the art forward).

It would be fun to use this for teaching CS3100 and also other OCaml tutorials. Perhaps even have an interactive version of Real World OCaml book.

Not all OCaml libraries can be compiled to JavaScript. The common reason being that they may depend on features not available on JavaScript. In writing this post, I unsuccessfully tried for a long time to get mirage-cypto working. mirage-crypto has a large C dependency, which does not work with Js_of_ocaml. Js_of_ocaml promises to take any opam library installed on your opam switch and compiles that to JavaScript. However, at that point, we’re really cross compiling the opam packages installed on your switch to JavaScript since the installed package may make some assumptions about the platform that it is supposed to run on. Hence, JavaScript compilation of arbitrary OCaml packages is unlikely to work in the general case. Unfortunately, the error was difficult to debug since the failure was at runtime, and was not apparent in the error messages (at least for me, who has little JavaScript experience). It would be nice to have the opam packages explicitly say whether they are JavaScript compatible, and have build tooling that reports errors like these early.

]]> KC Sivaramakrishnan https://kcsrk.info/ocaml/modes/oxcaml/2025/06/04/linearity_and_uniqueness/ https://kcsrk.info/ocaml/modes/oxcaml/2025/06/04/linearity_and_uniqueness/ Wed, 04 Jun 2025 10:00:00 GMT 'a t @ unique val free : 'a t @ unique -> unit val get : 'a t @ unique -> 'a Modes.Aliased.t * 'a t @ unique val set : 'a t @ unique -> 'a -> 'a t @ unique end Assume that we also have an implementation of the module: module Unique_ref : Unique_ref Consider the following example, which works fine: let works () = let t = alloc 42 in (* Allocate a unique reference *) free t (* free it *) Now consider this modified example: let wat () = let t = alloc 42 in (* Allocate a unique reference *) let f () = free t in (* capture free in a closure *) f (); (* free it *) f () (* free it again??? *) Observe that f has captured t in the closure, and when called frees t. It should be clear that calling f more than once is bad – leads to a double-free issue! What property do we want of f? Uniqueness is insufficient; we have a unique reference to f in this program, with which we call f twice. What we want to enforce is that f can be called at most once. The compiler has a linearity mode which captures the idea of how many times a value can be used. We have two modes in the linearity axis – once, which stands for “at most once” and many (the default one for all values), which allows values to be used arbitrary number of times. Whenever a unique value is captured by a closure, the closure gets a once mode, which allows the closure to be called at most once. This program rightly gets rejected by the compiler. File "./unique_ref.ml", line 32, characters 2-3: 32 | f () (* free it again??? *) ^ Error: This value is used here, but it is defined as once and has already been used: File "./unique_ref.ml", line 31, characters 2-3: 31 | f (); (* free it *) ^ A linear ref Now, one might wonder whether the unique reference that we’ve implemented may be implemented with the linear mode. The answer is yes. module type Linear_ref = sig type 'a t val alloc : 'a -> 'a t @ once val free : 'a t @ once -> unit val get : 'a t @ once -> 'a * 'a t @ once val set : 'a t @ once -> 'a -> 'a t @ once end module Linear_ref : Linear_ref = struct type 'a t = { mutable value : 'a } let alloc x = { value = x } let free t = () let get t = t.value, t let set t x = t.value <- x; t end This works as expected: open Linear_ref let works () = let r = alloc 42 in let v,r = get r in let r = set r (v + 1) in let v,r = get r in print_int v; free r; () let fails () = let r = alloc 42 in free r; get r (* fails here *) with the error message: File "./linear_ref.ml", line 34, characters 6-7: 34 | get r (* fails here *) ^ Error: This value is used here, but it is defined as once and has already been used: File "./linear_ref.ml", line 33, characters 7-8: 33 | free r; Why both linearity and uniqueness? Given this example, you might be wondering, if the safe reference may be implemented equivalently using both uniqueness and linearity, why do we need both? Obviously, there’s something interesting going on where unique values captured in a closure needs linearity. Does that mean linearity is sufficient? It turns out that only recently was the relationship between the two formally studied in the same type system. While linear types and uniqueness types have a long history of being studied independently, Marshall et al. in their paper, “Linearity and Uniqueness: An Entente Cordiale”, present the ideas in the same type system. They provide some key insights. The first insight is that in a setting where all values must be linear, we can also guarantee that every value is unique, and vice versa! Intuitively, if it is never possible to duplicate a value, then it will never be possible for said value to have multiple references. In our Unique_ref and Linear_ref every operation that operates on the ref requires uniqueness or linearity, respectively. Hence, they seem almost equivalent in expressive power. It is when we also have the ability for unrestricted use (non-linear/non-unique) that differences between linearity and uniqueness begin to arise, as we will soon see. In our language, we do have the ability for unrestricted use. That is, in the linearity axis, many is the default mode attributed to all the values not tagged or inferred as once. Similarly, aliased is the default mode attributed to all the values not tagged or inferred as unique. The type system has submoding: values may move freely to greater modes (which typically restrict what can be done with those values) but not to lesser modes. For example, a many value may be safely use in a context where a once value is expected. let works () = let set_to_20 (r @ once) = r := 20 in let r @ many = ref 10 in set_to_20 r (* [r @ many] is passed to a function that expects [int ref @ once] *) Similarly, you can use a unique value in a context where an aliased value is expected. let dup r = r,r let works () = let r = Unique_ref.alloc 42 in let a,b = dup r in a,b The type of the works function is val works : unit -> int Unique_ref.t * int Unique_ref.t, which crucially lacks the fact that the references are at unique mode. We can’t call any functions from the Unique_ref module with these references, all of which expect a reference with unique mode. Uniqueness is more appropriate for safe refs In our running example of implementing a safe ref, it turns out that uniqueness is more appropriate. Consider the type signature of free in Unique_ref: val free : 'a t @ unique -> unit The type signature says that there are no other aliases to this reference. Hence, its memory may be safely deallocated. However, consider the Linear_ref.free signature: val free : 'a t @ once -> unit The signature says that this reference must be used at most once. In particular, just by looking at the signature, we cannot conclude that there are no other aliases to this reference. But we know that the API is safe since the only way to create a safe reference is through the alloc function, which returns a once-usable reference, and every other operation also expects and returns a once-usable reference. The correctness of the linear version depends on reasoning over the whole API, whereas the unique version can be concluded to be safe just by looking at the signature of the free function. This modular reasoning makes uniqueness more appropriate for our safe reference API. Past and the future In a sense, uniqueness and linearity are duals of each other. Uniqueness talks about the past – whether a value may be aliased in the past. It is okay to alias a unique value in the future and lose the uniqueness mode. Linearity talks about the future – whether a value may be used more than once in the future. You can take any value and ascribe a linear mode to it, restricting its use in the future. However, there may be other aliases to this value in the past. Conclusions The code examples are available here. Section 2.1 of Marshall et al.’s paper is quite readable and explains the distinction between linearity and uniqueness with some historical context. I highly recommend it. Acknowledgements Thanks to Richard Eisenberg for the discussions which spurred this post.]]> In the last post, we looked at uniqueness mode and how uniqueness may be used to optimise. As we will see, uniqueness alone is insufficient in practice, and we also need a concept of linearity for uniqueness to be useful.

Capturing unique values

Let’s start with an example. Recall the signature of the unique reference module.

module type Unique_ref = sig
  type 'a t
  val alloc : 'a -> 'a t @ unique
  val free : 'a t @ unique -> unit
  val get : 'a t @ unique -> 'a Modes.Aliased.t * 'a t @ unique
  val set : 'a t @ unique -> 'a -> 'a t @ unique
end

Assume that we also have an implementation of the module:

module Unique_ref : Unique_ref

Consider the following example, which works fine:

let works () =
  let t = alloc 42 in (* Allocate a unique reference *)
  free t (* free it *)

Now consider this modified example:

let wat () =
  let t = alloc 42 in (* Allocate a unique reference *)
  let f () = free t in (* capture free in a closure *)
  f (); (* free it *)
  f () (* free it again??? *)

Observe that f has captured t in the closure, and when called frees t. It should be clear that calling f more than once is bad – leads to a double-free issue! What property do we want of f? Uniqueness is insufficient; we have a unique reference to f in this program, with which we call f twice.

What we want to enforce is that f can be called at most once. The compiler has a linearity mode which captures the idea of how many times a value can be used. We have two modes in the linearity axis – once, which stands for “at most once” and many (the default one for all values), which allows values to be used arbitrary number of times.

Whenever a unique value is captured by a closure, the closure gets a once mode, which allows the closure to be called at most once. This program rightly gets rejected by the compiler.

File "./unique_ref.ml", line 32, characters 2-3:
32 |   f () (* free it again??? *)
       ^
Error: This value is used here,
       but it is defined as once and has already been used:
File "./unique_ref.ml", line 31, characters 2-3:
31 |   f (); (* free it *)
       ^

A linear ref

Now, one might wonder whether the unique reference that we’ve implemented may be implemented with the linear mode. The answer is yes.

module type Linear_ref = sig
  type 'a t
  val alloc : 'a -> 'a t @ once
  val free : 'a t @ once -> unit
  val get : 'a t @ once -> 'a * 'a t @ once
  val set : 'a t @ once -> 'a -> 'a t @ once
end

module Linear_ref : Linear_ref = struct
  type 'a t = { mutable value : 'a }
  let alloc x = { value = x }
  let free t = ()
  let get t =
    t.value, t
  let set t x =
    t.value <- x;
    t
end

This works as expected:

open Linear_ref

let works () =
  let r = alloc 42 in
  let v,r = get r in
  let r = set r (v + 1) in
  let v,r = get r in
  print_int v;
  free r;
  ()

let fails () =
  let r = alloc 42 in
  free r;
  get r (* fails here *)

with the error message:

File "./linear_ref.ml", line 34, characters 6-7:
34 |   get r (* fails here *)
           ^
Error: This value is used here,
       but it is defined as once and has already been used:
File "./linear_ref.ml", line 33, characters 7-8:
33 |   free r;

Why both linearity and uniqueness?

Given this example, you might be wondering, if the safe reference may be implemented equivalently using both uniqueness and linearity, why do we need both? Obviously, there’s something interesting going on where unique values captured in a closure needs linearity. Does that mean linearity is sufficient?

It turns out that only recently was the relationship between the two formally studied in the same type system. While linear types and uniqueness types have a long history of being studied independently, Marshall et al. in their paper, “Linearity and Uniqueness: An Entente Cordiale”, present the ideas in the same type system. They provide some key insights.

The first insight is that

in a setting where all values must be linear, we can also guarantee that every value is unique, and vice versa! Intuitively, if it is never possible to duplicate a value, then it will never be possible for said value to have multiple references.

In our Unique_ref and Linear_ref every operation that operates on the ref requires uniqueness or linearity, respectively. Hence, they seem almost equivalent in expressive power.

It is when we also have the ability for unrestricted use (non-linear/non-unique) that differences between linearity and uniqueness begin to arise, as we will soon see.

In our language, we do have the ability for unrestricted use. That is, in the linearity axis, many is the default mode attributed to all the values not tagged or inferred as once. Similarly, aliased is the default mode attributed to all the values not tagged or inferred as unique.

The type system has submoding: values may move freely to greater modes (which typically restrict what can be done with those values) but not to lesser modes. For example, a many value may be safely use in a context where a once value is expected.

let works () =
  let set_to_20 (r @ once) =
    r := 20
  in
  let r @ many = ref 10 in
  set_to_20 r (* [r @ many] is passed to a function that expects [int ref @ once] *)

Similarly, you can use a unique value in a context where an aliased value is expected.

let dup r = r,r

let works () =
  let r = Unique_ref.alloc 42 in
  let a,b = dup r in
  a,b

The type of the works function is val works : unit -> int Unique_ref.t * int Unique_ref.t, which crucially lacks the fact that the references are at unique mode. We can’t call any functions from the Unique_ref module with these references, all of which expect a reference with unique mode.

Uniqueness is more appropriate for safe refs

In our running example of implementing a safe ref, it turns out that uniqueness is more appropriate. Consider the type signature of free in Unique_ref:

val free : 'a t @ unique -> unit

The type signature says that there are no other aliases to this reference. Hence, its memory may be safely deallocated. However, consider the Linear_ref.free signature:

val free : 'a t @ once -> unit

The signature says that this reference must be used at most once. In particular, just by looking at the signature, we cannot conclude that there are no other aliases to this reference. But we know that the API is safe since the only way to create a safe reference is through the alloc function, which returns a once-usable reference, and every other operation also expects and returns a once-usable reference.

The correctness of the linear version depends on reasoning over the whole API, whereas the unique version can be concluded to be safe just by looking at the signature of the free function. This modular reasoning makes uniqueness more appropriate for our safe reference API.

Past and the future

In a sense, uniqueness and linearity are duals of each other. Uniqueness talks about the past – whether a value may be aliased in the past. It is okay to alias a unique value in the future and lose the uniqueness mode. Linearity talks about the future – whether a value may be used more than once in the future. You can take any value and ascribe a linear mode to it, restricting its use in the future. However, there may be other aliases to this value in the past.

Conclusions

The code examples are available here. Section 2.1 of Marshall et al.’s paper is quite readable and explains the distinction between linearity and uniqueness with some historical context. I highly recommend it.

Acknowledgements

Thanks to Richard Eisenberg for the discussions which spurred this post.

]]> KC Sivaramakrishnan https://www.learningfromdata.zingg.ai/p/the-curious-case-of-the-extra-records https://www.learningfromdata.zingg.ai/p/the-curious-case-of-the-extra-records Sat, 31 May 2025 03:30:04 GMT

Earlier last month, we started working on a feature we are calling Match Statistics. The idea about statistics came from the incredible folks at Fortnum And Mason, who are using Zingg as the backbone for Single Customer View. While running Zingg incrementally, the Match Statistics would expose how clusters numbers change as records get inserted and updated into the identity graph. This information about changing clusters would be a good first step to observe the entity resolution pipeline. If the number of clusters change disproportionately to the number of records updated and added, an alert could be triggered.

As we started thinking about this feature, we realized that the Match Statistics feature could be made even more powerful. Currently the Zingg output consists of the original data along with the Zingg ID and the match probabilities. If we could surface information about the linkages we found in the records within the cluster, we could help users with matching internals and anomalies. When our users are dealing with millions of records, finding the needle in the haystack is critical. Hence, the specification for the feature expanded to additional statistics.

While building new features, or augmenting existing flows within Zingg, we have to design both the Community and Enterprise products, since they share the same code base. The design of any feature on either product has to take into consideration the impact on both products and ensure that they continue to work individually as well as in tandem. And then comes the trickier part. The design has to work well for both Snowflake and Spark deployments. Not just functional. Performance wise as well. The Zingg Enterprise code base is mostly common across Spark and Snowflake. However, since both platforms have very different APIs, query planning, optimisation and execution, we have a lot of ground to cover while building or modifying anything.

Thats what we thrive at :-)

Eventually we designed a stats package which would get the record level and cluster level matches from the match process. This package would be responsible for computing the metrics we cared about, and writing them to appropriate locations, or Pipes in Zingg lingo. The main flow would mostly be unaltered, except for invoking the stats package and passing the dataframes around.

As we started building the core functionality, we started seeing a very strange issue. Within our flow, we do probabilistic matching only on records which do not match deterministically. If two records match deterministically, we do not try to figure out if they match probabilistically as well since that is a waste of computation time.

Now, something strange started happening. Within the stats package, which was consuming the probabilistic and deterministic structures, we found some records which matched probabilistically as well as deterministically. This threw an error in the statistics computations, as they banked on the record pair to match one way or the other or not match at all. Our deeply crafted logic was breaking.

However, this part of the flow never changed. All we were doing was computing additional metrics in this package. And the main flow was unaltered. So how could stuff which was working earlier AND which was untouched stop working? What followed was a crazy cycle of runs, tests and validations.

Our initial thought was that we had made an error in collecting the data. Why not run it in another sandbox? Umm, the same result. Maybe we had run the flow twice appending to original intermittent data leading to the wrong state? But if that was the case, every record pair would appear twice. In our case, only some of the record pairs were being matched probabilistically as well as deterministically. We then sought to understand if there was anything special about those records. We looked at column values, deterministic rules, and the probabilistic match criteria. To our dismay, every run threw up a different set of extra records. But clearly, not data or code issue then.

Things were surely getting curiouser and curiouser. We were completely off track from the original feature we set out to build :-(

Next, we turned our focus to the execution environment. When we ran the build on Snowflake, there were no extra records. This was a relief, since it meant it was the Spark side of things that we needed to focus on. Now we ran the workload on Databricks. No issue at all! This was a relief too, since it meant earlier released production code deployed at customer locations was working fine and there was no snake in the grass.

After two weeks of racking our brains, we were finally making progress :-) Things started moving pretty fast post that. We searched for known issues and discovered a issue on Apache Spark. When we changed the Apache Spark version, the issue disappeared. To be absolutely sure, we tested different datasets, different volumes, and all looked good.

Debugging complex systems is always challenging, but some techniques always help.

1. Time travel to the last known good state helps understand which change or set of changes could be the cause.

2. Surgical elimination is super critical in such scenarios. Being able to quickly run different non-overlapping tests can lead to faster results.

3. Weird behavior - missing records, extra counts etc which alter at different runs are more likely to be caused by the environment than the actual code.

4. Such non deterministic errors usually stress teams out, since things feel very out of control. For engineers like us who are used to predictable outcomes, such issues can be very frustrating. Having a cool head goes a long way.

After this wild chase, we are now back to building the Match Statistics feature. We hope to ship it soon!

Just one last thing that I forgot to share. Many wise developers have said this before me, but I must say it again! Plan at least twice the time you anticipate it will take :-)

If you have a similar story to share, hit reply!

]]> Sonal Goyal https://kcsrk.info/ocaml/modes/oxcaml/2025/05/29/uniqueness_and_behavioural_types/ https://kcsrk.info/ocaml/modes/oxcaml/2025/05/29/uniqueness_and_behavioural_types/ Thu, 29 May 2025 17:56:00 GMT 'a t val free : 'a t -> unit (* unsafe *) val get : 'a t -> 'a val set : 'a t -> 'a -> unit end This interface provides an explicit free, which releases the memory associated with this reference. This opens up the possibility of memory safety bugs such as use-after-free and double-free. We can use uniqueness modality to get a safe API. Here’s the interface: module type S = sig type 'a t val alloc : 'a -> 'a t @ unique val free : 'a t @ unique -> unit val get : 'a t @ unique -> 'a Modes.Aliased.t * 'a t @ unique val set : 'a t @ unique -> 'a -> 'a t @ unique end The unique annotation states that the value is not aliased. The operations on the reference expect that this reference is not aliased. Observe that get and set take in the unique reference and also return them unlike the original interface. You can use this like so: # let okay r = let v, r = get r in let r = set r 20 in free r;; val okay : int M.t @ unique -> unit = The key bit is that free consumes the unique reference; you can no longer produce a unique handle to the same reference and hence, you cannot call free, get or set on this reference which has been freed. # let wont_work r = free r; get r ;; Error: This value is used here, but it has already been used as unique: Line 2, characters 7-8: Modes.Aliased.t Uniqueness applies deeply. If a value is marked as unique, then the transitive closure of the reachable parts of the object is also expected to be unique. The return value of get is a pair, which is marked as unique1. Hence, both the components of the pair are expected to be unique. However, we don’t want to impose uniqueness of the value stored in the reference. The language allows parts of the value to be marked as aliased. Modes.Aliased.t is defined as: module Aliased : sig type 'a t = { aliased : 'a @@ aliased } [@@unboxed] end The language allows record fields to be annotated as aliased, while the record itself may be uniquely referenced. Implementation Here’s an implementation of that satisfies the signature. module M : S = struct type 'a t = { mutable value : 'a } let alloc x = { value = x } let free t = () let get t = let a = Modes.Aliased.{aliased = t.value } in a, t let set t x = t.value <- x; t end There’s nothing surprising about this implementation. Note that the compiler is doing a lot of work behind the scenes to ensure that the functions do in fact satisfy the uniqueness requirements. For example, if you change the implementation of set to do something innocuous where the compiler cannot prove that the value is not aliased, the program no longer compiles: # let set t x = t.value <- x; let t' = Fun.id t in (* compiler cannot prove [t'] is not aliased *) t' Error: Values do not match: val set : 'a t -> 'a -> 'a t is not included in val set : 'a t @ unique -> 'a -> 'a t @ unique The type 'a t -> 'a -> 'a t is not compatible with the type 'a t @ unique -> 'a -> 'a t @ unique Refs that explain their work The earlier blog post used polymorphic variants to encode the protocol of operations that are permitted on a ref cell. The implementation is reproduced below: module type Ref = sig type ('a, 'b) ref constraint 'b = [>] val ref : 'a -> ('a, 'b) ref val read : ('a, [`Read of 'b]) ref -> 'a * ('a, 'b) ref val write : ('a, [`Write of 'b]) ref -> 'a -> ('a, 'b) ref end module Ref : Ref = struct type ('a, 'b) ref = {contents : 'a; mutable live : bool} (* For linearity *) constraint 'b = [>] let ref v = {contents = v; live = true} let check r = if not r.live then raise LinearityViolation; r.live <- false let fresh r = {r with live = true} let read r = check r; (r.contents, fresh r) let write r v = check r; { contents = v; live = true } let branch r _ = check r; fresh r end Observe that we use a dynamic check to enforce linearity. It requires a fresh ref cell for each operation performed on this reference. With uniqueness, we can enforce this statically, avoiding the dynamic check and the fresh ref cell requirement. module type Ref = sig type ('a, 'b) ref constraint 'b = [>] (* 'b is the behavioural type variable *) val ref : 'a -> ('a, 'b) ref @ unique val read : ('a, [`Read of 'b]) ref @ unique -> 'a Modes.Aliased.t * ('a, 'b) ref @ unique val write : ('a, [`Write of 'b]) ref @ unique -> 'a -> ('a, 'b) ref @ unique val branch : ('a, [>] as 'b) ref @ unique -> (('a, [>] as 'c) ref @ unique -> 'b) -> ('a, 'c) ref @ unique end module Ref : Ref = struct type ('a, 'b) ref = {mutable contents : 'a} constraint 'b = [>] let ref v = {contents = v} let read r = let c = Modes.Aliased.{aliased = r.contents} in c, Obj.magic_at_unique r let write r v = r.contents <- v; Obj.magic_at_unique r let branch r _ = Obj.magic_at_unique r end The only changes necessary in the signature were a number of uniqueness and aliasing annotations. Notice that the implementation no longer needs the dynamic check! Obj.magic_at_unique has the type 'a @ unique -> 'b @ unique, and is the version of Obj.magic with uniqueness annotation. We use it to advance the protocol type state. Where next The rest of the examples in the original post should also benefit from uniqueness annotations to remove the runtime overheads. The complete code examples are available here. You can also play with the code examples directly in the browser thanks to Patrick Ferris’ OCaml with extensions js_of_ocaml top-level. Since the modes features are constantly evolving, there are no stability guarantees yet. However, I’m excited about the possibility of modes improving how we do safe systems programming in OCaml. Addendum Looks like there’s a part 2 of this post. Footnotes Unclear whether it is possible to return a pair where one of the components is unique, but the other one is not. ↩]]> Jane Street has been developing modal types for OCaml – an extension to the type system where modes track properties of values, such as their scope, thread sharing, and aliasing. These modes restrict which operations are permitted on values, enabling safer and more efficient systems programming. In this post, I focus on the uniqueness mode, which tracks aliasing, and show how it can eliminate certain runtime checks.

My intention in this post is not to explain how the different modes work. There are a number of blog posts and academic papers written about modes. I recommend the interested reader to have look at them. The following table summarizes the main properties tracked by modes, the corresponding mode names, and resources for further reading:

Property	Modes	Resources
Scope	Locality	Blog, Paper
Sharing between threads	Portability, Contention	Blog, Paper
Aliasing	Uniqueness, Linearity	Blog, Paper

The OCaml compiler extended with modes is developed in the open, and is used in production at Jane Street. The repo also has some documentation of the extensions.

Be warned that the compiler and the language features are fast evolving. The code examples presented in the blog and the paper referenced above are likely not to work. I expect the same for the code examples in this post in the near future, but that’s what one should expect with these bleeding-edge features.

Behavioural types and runtime overhead

A couple of years ago, I wrote a post on behavioural types where the types capture the sequence of operations that may be performed on the values with those types. The correctness of the system depended on the linear use of the resources. Since OCaml does not provide support for enforcing linearity statically, the implementation uses a dynamic check, using a fresh ref cell that gets consumed every time the type state changes. If we are guaranteed that the resource is not aliased statically, then there’s no need for the dynamic check. This is where uniqueness helps.

Uniqueness mode allows the OCaml compiler to statically guarantee that certain values are not aliased. This enables optimizations and eliminates the need for some runtime checks, which is particularly valuable in systems programming for ensuring memory safety and efficient resource management.

Setting up OCaml with modes

An opam repository with the modes extensions and packages supporting modes is available here. Here’s how you can set up the new compiler:

# this will take time
opam switch create 5.2.0+flambda2 --repos with-extensions=git+https://github.com/janestreet/opam-repository.git#with-extensions,default
eval $(opam env --switch 5.2.0+flambda2)

An explicitly memory-managed reference

Suppose you want to implement a mutable reference whose memory is explicitly managed (not managed by the GC), you may go for the following interface:

module type S = sig
  type 'a t
  val alloc : 'a -> 'a t
  val free : 'a t -> unit (* unsafe *)
  val get : 'a t -> 'a
  val set : 'a t -> 'a -> unit
end

This interface provides an explicit free, which releases the memory associated with this reference. This opens up the possibility of memory safety bugs such as use-after-free and double-free. We can use uniqueness modality to get a safe API. Here’s the interface:

module type S = sig
  type 'a t
  val alloc : 'a -> 'a t @ unique
  val free : 'a t @ unique -> unit
  val get : 'a t @ unique -> 'a Modes.Aliased.t * 'a t @ unique
  val set : 'a t @ unique -> 'a -> 'a t @ unique
end

The unique annotation states that the value is not aliased. The operations on the reference expect that this reference is not aliased. Observe that get and set take in the unique reference and also return them unlike the original interface. You can use this like so:

# let okay r =
    let v, r = get r in
    let r = set r 20 in
    free r;;
val okay : int M.t @ unique -> unit = <fun>

The key bit is that free consumes the unique reference; you can no longer produce a unique handle to the same reference and hence, you cannot call free, get or set on this reference which has been freed.

# let wont_work r =
    free r;
    get r
  ;;
Error: This value is used here, but it has already been used as unique:
Line 2, characters 7-8:

Modes.Aliased.t

Uniqueness applies deeply. If a value is marked as unique, then the transitive closure of the reachable parts of the object is also expected to be unique. The return value of get is a pair, which is marked as unique¹. Hence, both the components of the pair are expected to be unique. However, we don’t want to impose uniqueness of the value stored in the reference. The language allows parts of the value to be marked as aliased. Modes.Aliased.t is defined as:

module Aliased : sig
  type 'a t = { aliased : 'a @@ aliased } [@@unboxed]
end

The language allows record fields to be annotated as aliased, while the record itself may be uniquely referenced.

Implementation

Here’s an implementation of that satisfies the signature.

module M : S = struct
  type 'a t = { mutable value : 'a }
  let alloc x = { value = x }
  let free t = ()
  let get t =
    let a = Modes.Aliased.{aliased = t.value } in
    a, t
  let set t x =
    t.value <- x;
    t
end

There’s nothing surprising about this implementation. Note that the compiler is doing a lot of work behind the scenes to ensure that the functions do in fact satisfy the uniqueness requirements. For example, if you change the implementation of set to do something innocuous where the compiler cannot prove that the value is not aliased, the program no longer compiles:

# let set t x =
    t.value <- x;
    let t' = Fun.id t in (* compiler cannot prove [t'] is not aliased *)
    t'
Error: <snip>
Values do not match:
 val set : 'a t -> 'a -> 'a t
is not included in
 val set : 'a t @ unique -> 'a -> 'a t @ unique
The type 'a t -> 'a -> 'a t is not compatible with the type
 'a t @ unique -> 'a -> 'a t @ unique

Refs that explain their work

The earlier blog post used polymorphic variants to encode the protocol of operations that are permitted on a ref cell. The implementation is reproduced below:

module type Ref =
sig
  type ('a, 'b) ref constraint 'b = [>]

  val ref   : 'a -> ('a, 'b) ref
  val read  : ('a, [`Read of 'b]) ref
              -> 'a * ('a, 'b) ref
  val write : ('a, [`Write of 'b]) ref
              -> 'a
              -> ('a, 'b) ref
end
module Ref : Ref =
struct

  type ('a, 'b) ref =
    {contents     : 'a;
     mutable live : bool} (* For linearity *)
     constraint 'b = [>]

  let ref v = {contents = v; live = true}

  let check r =
    if not r.live then raise LinearityViolation;
    r.live <- false

  let fresh r = {r with live = true}

  let read r =
    check r;
    (r.contents, fresh r)

  let write r v =
    check r;
    { contents = v; live = true }

  let branch r _ = check r; fresh r
end

Observe that we use a dynamic check to enforce linearity. It requires a fresh ref cell for each operation performed on this reference. With uniqueness, we can enforce this statically, avoiding the dynamic check and the fresh ref cell requirement.

module type Ref =
sig
  type ('a, 'b) ref constraint 'b = [>]
  (* 'b is the behavioural type variable *)

  val ref   : 'a -> ('a, 'b) ref @ unique
  val read  : ('a, [`Read of 'b]) ref @ unique
              -> 'a Modes.Aliased.t * ('a, 'b) ref @ unique
  val write : ('a, [`Write of 'b]) ref @ unique
              -> 'a
              -> ('a, 'b) ref @ unique
  val branch : ('a, [>] as 'b) ref @ unique
               -> (('a, [>] as 'c) ref @ unique -> 'b)
               -> ('a, 'c) ref @ unique
end

module Ref : Ref =
struct
  type ('a, 'b) ref = {mutable contents : 'a} constraint 'b = [>]

  let ref v = {contents = v}

  let read r =
    let c = Modes.Aliased.{aliased = r.contents} in
    c, Obj.magic_at_unique r

  let write r v =
    r.contents <- v;
    Obj.magic_at_unique r

  let branch r _ = Obj.magic_at_unique r
end

The only changes necessary in the signature were a number of uniqueness and aliasing annotations. Notice that the implementation no longer needs the dynamic check! Obj.magic_at_unique has the type 'a @ unique -> 'b @ unique, and is the version of Obj.magic with uniqueness annotation. We use it to advance the protocol type state.

Where next

The rest of the examples in the original post should also benefit from uniqueness annotations to remove the runtime overheads.

The complete code examples are available here. You can also play with the code examples directly in the browser thanks to Patrick Ferris’ OCaml with extensions js_of_ocaml top-level.

Since the modes features are constantly evolving, there are no stability guarantees yet. However, I’m excited about the possibility of modes improving how we do safe systems programming in OCaml.

Addendum

Looks like there’s a part 2 of this post.

Footnotes

1. Unclear whether it is possible to return a pair where one of the components is unique, but the other one is not. ↩

]]> KC Sivaramakrishnan https://ravidwivedi.in/posts/singapore-visa/ https://ravidwivedi.in/posts/singapore-visa/ Tue, 27 May 2025 14:50:49 GMT In November 2024, Badri and I applied for a Singapore visa to visit the country. To apply for a Singapore visa, you need to visit an authorized travel agent listed by the Singapore High Commission on their website. Unlike the Schengen visa (where only VFS can process applications), the Singapore visa has many authorized travel agents to choose from. I remember that the list mentioned as many as 25 authorized agents in Chennai. For my application, I randomly selected Ria International in Karol Bagh, New Delhi from the list.

Further, you need to apply not more than a month before your travel dates. As our travel dates were in December, we applied in the month of November.

For your reference, I submitted the following documents:

• Passport
• My photograph (35 mm x 45 mm)
• Visa application form (Form 14A)
• Cover letter to the Singapore High Commission, New Delhi
• Proof of employment
• Hotel booking
• Flight ticket (reservations are sufficient)
• Bank account statement for the last 6 months

I didn’t have my photograph in the specified dimensions, so the travel agent took my photo on the spot. The visa application was ₹2,567. Furthermore, I submitted my application on a Saturday and received a call from the travel agent on Tuesday informing me that they had received my visa from the Singapore High Commission.

The next day, I visit the travel agent’s office and picked up my passport and a black and white copy of my e-visa. Later, I downloaded a PDF of my visa from the website mentioned on it, and took a colored printout myself.

Singapore granted me a multiple-entry visa for 2 months, even though I had applied for a 4-day single-entry visa. We were planning to add more countries to this trip; therefore, a multiple-entry visa would be helpful in case we wanted to use Singapore Airport, as it has good connectivity. However, it turned out that flights from Kuala Lumpur were much cheaper than those from Singapore, so we didn’t enter Singapore again after leaving.

Badri also did the same process but entirely remotely—he posted the documents to the visa agency in Chennai, and got his e-visa in a few days followed by his original passport which was delivered by courier.

He got his photo taken in the same dimensions mentioned above, and printed as matte finish as instructed. However, the visa agents asked why his photo was looking so faded. We don’t know if they thought the matte finish was faded or what. To rectify this, Badri emailed them a digital copy of the photo to them (both the cropped version and the original) and they handled the reprinting on their end (which he never got to see).

Before entering Singapore, we had to fill an arrival card - an online form asking a few details about our trip - within 72 hours of our arrival in Singapore.

That’s it for now. Meet you in the next post.

Thanks to Badri for reviewing the draft.

]]> Ravi Dwivedi https://workdone0.substack.com/p/national-anthem https://workdone0.substack.com/p/national-anthem Sat, 24 May 2025 00:00:00 GMT Shubham Kumar https://nadh.in/blog/mcp-seems-viral/ https://nadh.in/blog/mcp-seems-viral/ Mon, 19 May 2025 00:00:00 GMT MCP (Model Context Protocol)^[1] is all the rage now. Introduced by Anthropic about four months ago, it has already been accepted as an open standard and has seen widespread adoption, including by major AI companies and prominent AI tool makers. There is even a directory^[2] listing over 13,000 MCP implementations. Technically, it is a very simple API spec that facilitates RPC-like (Remote procedure call) communication between an AI and an external system. It enables any external system to advertise its capabilities—returning information or performing various actions—allowing AI systems to dynamically and “automagically” use those capabilities via API calls.

]]> Kailash Nadh https://ravidwivedi.in/posts/kde-india-conference-2025/ https://ravidwivedi.in/posts/kde-india-conference-2025/ Tue, 13 May 2025 17:58:18 GMT Last month, I attended the KDE India conference in Gandhinagar, Gujarat from the 4th to the 6th of April. I made my mind to attend when Sahil told me about his plans to attend and giving a talk.

A day after my talk submission, the organizer Bhushan contacted me on Matrix and informed me that my talk had been accepted. I was also informed that KDE will cover my travel and accommodation expenses. So, I planned to attend the conference at this point. I am a longtime KDE user, so why not ;)

I arrived in Ahmedabad, the twin city of Gandhinagar, a day before the conference. The first thing that struck me as soon as I came out of the Ahmedabad airport was the heat. I felt as if I was being cooked—exactly how Bhushan put it earlier in the group chat. I took a taxi to get to my hotel, which was close to the conference venue.

Later that afternoon, I met Bhushan and Joseph. Joseph lived in Germany. Bhushan was taking him to get a SIM card, so I tagged along and got to roam around. Joseph was unsure about where to go after the conference, so I asked him what he wanted out of his trip and had conversations along that line.

Later, Vishal convinced him to go to Lucknow. Since he was adamant about taking the train, I booked a Tatkal train ticket for him to Lucknow. He was curious about how Tatkal booking works and watched me in amusement while I was booking the ticket.

The 4th of April marked the first day of the conference, with around 25 attendees. Bhushan started the day with an overview of KDE conferences in India, followed by Vishal, who discussed FOSS United’s activities. After the lunch, Joseph gave an overview of his campaign to help people switch from Windows to GNU/Linux due to environmental and security reasons. He continued his session in detail the next day.

A key takeaway for me from Joseph’s session was the idea pointed out by Adwaith: marketing GNU/Linux as a cheap alternative may not attract as much attention as marketing it as a status symbol. He gave the example of how the Tata Nano didn’t do well in the Indian market due to being perceived as a poor person’s car.

My talk was scheduled for the evening of the first day. I hadn’t prepared any slides because I wanted to make my session interactive. During my talk, I did an activity with the attendees to demonstrate the federated nature of XMPP messaging, of which Prav is a part. After the talk, I got a lot of questions, signalling engagement. The audience was cooperative (just like Prav ;)), contrary to my expectations (I thought they will be tired and sleepy).

On the third day, I did a demo on editing OpenStreetMap (referred to as “OSM” in short) using the iD editor. It involved adding points to OSM based on the students’ suggestions. Since my computer didn’t have an HDMI port, I used Subin’s computer, and he logged into his OSM account for my session. Therefore, any mistakes I made will be under Subin’s name. :)

On the third day, I attended Aaruni’s talk about backing up a GNU/Linux system. This was the talk that resonated with me the most. He suggested formatting the system with the btrfs file system during the installation, which helps in taking snapshots of the system and provides an easy way to roll back to a previous version if, for example, a file is accidentally deleted. I have tried many backup techniques, including this one, but I never tried backing up on the internal disk. I’ll certainly give this a try.

A conference is not only about the talks, that’s why we had a Prav table as well ;) Just kidding. What I really mean is that a conference is more about interactions than talks. Since the conference was a three-day affair, attendees got plenty of time to bond and share ideas.

After the conference, Bhushan took us to Adalaj Stepwell, an attraction near Gandhinagar. Upon entering the complex, we saw a park where there were many langurs. Going further, there were stairs that led down to a well. I guess this is why it is called a stepwell.

Later that day, we had Gujarati Thali for dinner. It was an all-you-can-eat buffet and was reasonably priced at 300 rupees per plate. Aamras (Mango juice) was the highlight for me. This was the only time we had Gujarati food during this visit. After the dinner, Aaruni dropped Sahil and I off at the airport. The hospitality was superb - for instance, in addition to Aaruni dropping us, Bhushan also picked up some of the attendees from the airport.

Finally, I would like to thank KDE for sponsoring my travel and accommodation costs.

Let’s wrap up this post here and meet you in the next one.

Thanks to contrapunctus and Joseph for proofreading.

]]> Ravi Dwivedi https://www.learningfromdata.zingg.ai/p/what-is-in-a-name https://www.learningfromdata.zingg.ai/p/what-is-in-a-name Mon, 12 May 2025 06:39:09 GMT When Shakespeare penned these famous words, he wasn't thinking about data quality or entity resolution—but the sentiment captures a fundamental challenge in modern data management. A person known as "Robert," "Rob," "Bob," or "Bobby" remains the same individual, regardless of how they're addressed in the data. Names are perhaps the most fundamental identifier we use in data systems, yet they're also among the most inconsistent. Let us consider these common scenarios:

• Nicknames and diminutives: William becomes Will, Bill, or Billy

• Cultural variations: James in English is Diego in Spanish or Jacques in French

• Formal vs. informal usage: Margaret on official documents, Maggie among friends

• Transliterations: Китай (Russian) becomes Kitay in Latin script

• Spelling inconsistencies: Steven vs. Stephen, Catherine vs. Katherine

These variations create significant challenges for data matching. Without proper handling, our systems might fail to recognize that Robert Smith and Bob Smith are the same person, leading to duplicated records, fragmented customer views, and inaccurate analytics. For customer-facing applications, recognizing these are the same person enables more personalized service and prevents frustrating redundancies. Regulatory frameworks require comprehensive entity resolution. Proper nickname handling helps ensure compliance with KYC (Know Your Customer), AML (Anti-Money Laundering), and GDPR requirements. False negatives—failing to match records that should be matched—can be particularly costly in fraud detection, risk and healthcare.

While it might seem straightforward to compare names for exact matches, the reality is far more complex. We already handle salutations and variations in the data. Yet, without nickname handling, many true matches go undetected. Our internal benchmarks showed that incorporating nickname matching can improve overall match rates by 10-25% in typical customer datasets.

Hence, it was time to build! Amongst the numerous approaches we considered, dictionary matching shone out for a couple of reasons.

• Name variations differ significantly across cultures. Dictionaries can be configured to prioritize certain cultural naming patterns based on the data demographics. We are doing Hebrew names in one case and English names in another.

• Configurable dictionaries empower users to leverage their domain knowledge for entity resolution.

• Even without user involvement, a nickname dictionary mapping formal names to their common variations is not that difficult to build in the age of LLMs.

• The surrounding data points of the record like address, date of birth, contact information) can be easily validated for potential nickname matches to reduce false positives.

• Different industries and use cases require different sensitivity levels. Healthcare applications might prioritize precision, while marketing applications might favor recall. A configurable dictionary helps in such cases.

As we zeroed in on the dictionary, we realized that the approach is extensible to other domains like corporations, where mapping an IBM to International Business Machine will greatly improve matching. In the end, we built a MAPPING match type that can even normalize categorical columns like gender, or states and countries.

Rolling new features on running production pipelines is not trivial. For existing users, we did comprehesive A/B testing of entity resolution with and without nickname handling to quantify the improvement in their specific datasets. The results have been even better than we expected!

Names are at once the most essential and most variable identifiers in our data systems. Effective entity resolution must account for the rich complexity of how names are used in the real world. As Shakespeare suggested, a rose by any other name would indeed smell as sweet—and with Zingg's nickname matching, data systems will finally recognize that truth!

]]> Sonal Goyal https://shrirangkahale.com/posts/encrypted-adns/ https://shrirangkahale.com/posts/encrypted-adns/ Sun, 11 May 2025 09:43:19 GMT Shrirang Kahale https://ravidwivedi.in/posts/paris/ https://ravidwivedi.in/posts/paris/ Mon, 05 May 2025 20:02:43 GMT After attending the 2024 LibreOffice conference in Luxembourg, I visited Paris in October 2024.

If you are wondering whether I needed another visa to cross the border into France— I didn’t! Further, they are both also EU members, which means you don’t need to go through customs either. Thus, crossing the Luxembourg-France border is no different from crossing Indian state borders - like going from Rajasthan to Uttar Pradesh.

I took a TGV train from Luxembourg Central Station, which was within walking distance from my hostel. The train took only 2 hours and 20 minutes to cover the 300 km distance to Paris. It departed from Luxembourg at 10:00 AM and reached Paris at 12:20 PM. The ride was smooth and comfortable, arriving on time. It gave me an opportunity to see the countryside of France. I booked the train ticket online a couple of days prior through the Omio website.

I planned the first day with my friend Joenio, whom I met upon arriving in Paris’ Gare de l’Est station, along with his wife Mari. We went to my hostel (which was within walking distance from the station) to store my luggage, but we were informed that we needed to wait for a couple of hours before I could check in. Consequently, we went to an Italian restaurant nearby for lunch, where I ordered pasta. My hostel was unbelievably cheap by French standards (25 euros per night) that Joenio was shocked when he learned about it.

Walking in the city, I noticed it had separate cycling tracks and wide footpaths, just like Luxembourg. The traffic was also organized. For instance, there were traffic lights even for pedestrian crossings, unlike India, where crossing roads can be a nightmare. Car drivers stopping for pedestrians is a big improvement over what I am used to in India. The weather was also pleasant. It was a bit on the cooler side - around 15 degrees Celsius - and I had to wear a jacket.

After lunch, we returned to my hostel for my check-in at around 3 o’clock. Then, we went to the Luxembourg Museum (Musée du Luxembourg in French) as Joenio had booked tickets for an exhibition of paintings by the Brazilian painter Tarsila do Amaral. To reach there, we took a subway train from Gare du Nord station. The Paris subway charges 2.15 euros irrespective of the distance (or number of stations) traveled, as opposed to other metro systems I have used.

We reached the museum at around 4 o’clock. I found the paintings beautiful, but I would have appreciated them much more if the descriptions were in English.

Afterward, we went to a beautiful garden just behind the museum. It served as a great spot to relax and take pictures. Following this, we walked to the Pantheon - a well-known attraction in the city. It is a church built a couple of centuries ago. It has a dome-shaped structure at the top, recognizable from far away.

Then we went to Notre Dame after having evening snacks and coffee at a nearby bakery. The Notre Dame was just over a kilometer from the Pantheon, so we took a walk. We also crossed the beautiful Seine river. On the way, I sampled a crêpe, a signature dish of France. The shop was named Crêperie and had many varieties of Crêpe. I took the one with eggs and Emmental cheese. It was savory and delicious.

By the time we reached Notre Dame, it was 07:30 PM. I learned from Joenio that Notre Dame was closed and being renovated due to a fire a couple of years ago, so we just sat around and clicked photos. It is a catholic cathedral built in French Gothic architecture (I read that on Wikipedia ;)). I read on Wikipedia that it is located on an island named Île de la Cité and I didn’t even realize we are on an island.

At night, we visited the most well-known attraction of Paris, The Eiffel Tower. We again took the subway, alighting at the Bir-Hakeim station, followed by a short walk. We reached the Eiffel Tower at 9 o’clock. It was lit bright yellow. There was not much to do there, so we just clicked photos and hung out. After that, I came back to my hostel.

Next day, I roamed around the city by walking mostly. France is known for its bakeries, so I checked out a couple of local bakeries. I had espresso a couple of times and sampled croissant, pain au chocolat and lemon meringue tartlet.

Here are some random shots:

On the third day, I had my flight for India. Thus, I checked out of the hostel early in the morning, took an RR train from Gare du Nord station to reach the airport. It costs 11.8 euros.

I am listing my expenses during my 3-day stay in Paris below:

Category	Amount (in euros)
Food	€40
Public Transport	€20
Accommodation (2 nights)	€50
Total	€110

I heard some of my friends had bad experiences in France. Thus, I had the impression that I would not feel welcomed. Furthermore, I have encountered language problems during my previous Europe trip in Albania and Kosovo. Likewise, I learned a couple of French words, like how to say thank you and good morning, which went a long way.

However, I didn’t have bad experiences in Paris, except for one instance in which I asked my hostel’s reception about my misplaced watch and the person at the reception asked me to be “polite” by being rude. She said, “Excuse me! You don’t know how to say Good Morning?”

Overall, I enjoyed my time in Paris and would like to thank Joenio and Mari for showing me around. I would also like to thank Sophie for giving me a map of Paris, which was handy.

Let’s end this post here. I’ll meet you in the next one!

Credits: Thanks to contrapunctus for reviewing this post before publishing

]]> Ravi Dwivedi https://kcsrk.info/ocaml/iitm/community/2025/04/28/working-with-me/ https://kcsrk.info/ocaml/iitm/community/2025/04/28/working-with-me/ Mon, 28 Apr 2025 12:10:00 GMT Recently, I posted on X and LinkedIn that I am always looking for excellent people to join my group. I received a lot of enquiries, some of which led to internship hires (yay!). But mostly, I seemed to offer similar advice. I thought I’d write a post that summarise my responses.

At IIT Madras, my research group develops programming language abstractions to solve systems problems. The group is composed of research associates (fixed-term project staff), PhD, MS and MTech students, undergraduate research students (who are typically BTech students from IIT Madars) and interns. I made the following post a few weeks ago, for which I received a lots of enquiries, and I have been busy writing similar responses to many of them, which I summarise below.

PSA: I'm always looking for excellent folks to join my research group at IIT Madras to work on building "functional" systems. This includes internships, MS and PhD studentships, research staff positions, and post-baccalaureate fellowships.

Reach out to me if you are keen!

— KC Sivaramakrishnan (@kc_srk) April 15, 2025

Internship positions

Internship enquiries are the most frequent ones that I receive. Here’s how you can make it work. Please do go through my web page to look at what areas I work on. Write to me about what interests you and what your goals are.

My group works on systems. To make the internship work well, we require that you have demonstrable systems building experience. Do build projects that go beyond your coursework. Make sure that the projects are developed publicly on GitHub or other similar platforms so that one can take a look at what you’ve built. Even better is contributions to other open-source projects.

The group solves systems problems with functional programming. If you have prior experience with functional programming, such as building small projects with OCaml, Haskell, Scala, Scheme or other languages, it is easier for me to assess your interest. That said, if you are great at any programming language, having built non-trivial projects in any language, then you have the right skills for internships in my group. Generally, I expect the interns to have done course work on OS, compilers and computer architecture. Significant projects in any of those areas is a huge plus.

I should clarify that my recommendation letters for graduate programs will reflect my honest assessment of the internship. I will decline writing a recommendation letter if I think I may not be able to provide a strong one.

I do not work on projects that are primarily AI/ML or Web Development. If you write to me looking for projects in those areas, it is very likely that you won’t hear from me. Please don’t bulk email faculty CCing or BCCing everyone in the department. It is likely that no one will read such an email.

PhD/MS/MTech positions

For academic positions, please have a look at https://research.iitm.ac.in/. There are alternative ways to enter MS and PhD positions by being a reserach associate and completing some coursework at IITM. For more information, see here.

Contributing to the OCaml community

A significant chunk of the enquiries were from folks who hold full-time positions looking to be involved in the research group. Unfortunately, making part-time positions work is a challenge for both sides. I would encourage contributions to the wider OCaml community.

There are several great ways to get involved with the community. Here’s what I usually recommend.

• Learn the basics.
  • Go through the OCaml part of my CS3100 course. The course has a YouTube playlist and programming assignments. Complete the programming assignments.
  • Read the Real World OCaml book.
  • There are lots of other resources at OCaml.org, the official website of the OCaml community and the ecosystem.
• Join the community.
  • OCaml discord and discuss are great places to hang out with other OCaml folks and ask questions.
  • Discord is better for quick clarifications and discuss for longer form discussions.
• Look for “good first issues” in the OCaml projects and work on them
  • Check out the core platform tools under the OCaml github org. See OCaml compiler, dune build system, opam package manager, ocaml.org, etc.
  • Across the wider ecosystem – SemGrep, OpenGrep, Rocq, etc.
• Work on self-directed projects. Here is my list of ideas.

OCaml community also participates in Outreachy internships. Outreachy internships are paid internships for underrepresented groups. It is a great way to contribute to the community while being mentored by folks from the OCaml community. Here’s a nice intro (in Tamil) to the impact that Outreachy program had on an Outreachy intern. Look out for announcements about Outreachy internships in the OCaml discuss forum.

Research Associate positions

This is for folks who want to contribute to the core research programme but do not see themselves joining academic programs. The expectation here is that you are an experiened systems engineer, who should see themselves easily qualifying for the internship positions in the group.

One useful way to look at this position is similar to a research software development engineer who helps build out the systems used for research or translate research to practice. In the past, research associates have helped upstream multicore OCaml. The easiest way to get into this role would be to do an internship, see whether you like this area, do well in the internship and then choose to apply to research associate position.

Another variant is a post-bacc or a pre-doc position aimed at highly motivated recent graduates, who are looking to build research experience. The expectation here is that we get papers into top venues in PL and Systems. For such students, I recommend going through my CS6225 Programs and Proofs course, watch the video lectures and complete the assignments. The course is not an easy one, but will expose you to the broad area of PL and specifically to deductive program verification. At the very least, you will come out with an understanding of what it is to think rigorously about program correctness.

Research associate positions are fixed-term positions. In order to make this work, the tenure should be at least 18 months to make it work.

Summary

While I may not be hiring actively all the time, do reach out to me if you are interested in any of hte above. Please follow me on LinkedIn, X or Bluesky, where I am likely to announce any open positions.

]]> KC Sivaramakrishnan https://mrkaran.dev/posts/announcing-logchef/ https://mrkaran.dev/posts/announcing-logchef/ Sun, 27 Apr 2025 00:00:00 GMT

Demo Credentials:
Username: demo@logchef.app
Password: password

What’s Next?# Logchef is already being used internally, but the journey towards a full v1.0 release continues this year. The roadmap includes exciting additions like: Alerting: Trigger notifications based on query results. Live Tail Logs: Stream logs in real-time. Enhanced Dashboarding: More powerful visualization capabilities. Logchef is open source (AGPLv3), and community involvement is welcomed. You can check out the Demo or view the code on GitHub. If you have more ideas or features you’d like to see, please reach out on GitHub or email me! I’m always open to suggestions and feedback. Honestly, building Logchef has been incredibly rewarding. It started as a way to fix something that bugged me (and others!), and seeing it turn into a tool I’m genuinely excited about feels great. I couldn’t have done it alone, though. I’m really grateful to my friends and colleagues who jumped in with feedback along the way. Huge thanks to Kailash for the constant support and encouragement, and to Vivek, Sarat, and Rohan for testing the early builds and offering great suggestions. Finally, a big thank you to my wife, who patiently endured my late-night coding sessions. Her support means the world to me <3 Fin!]]> So, for the last 3-4 months, I’ve been busy building Logchef. This tool basically grew straight out of my day job managing logs at Zerodha, where I’ve been managing logs for almost half a decade. I wanted to share a bit about how Logchef came to be.

Like many, we journeyed through the complexities of ELK (a management nightmare) and found its OSS fork, OpenSearch, didn’t quite hit the mark for us either. We eventually found solid ground with Clickhouse, as detailed on our tech blog: Logging at Zerodha.

Challenges Faced with Metabase#

However, as I noted in that post, while Metabase served us well for analytics, it wasn’t the ideal UI specifically tailored for log analysis against Clickhouse:

“While Metabase has served us well so far, there is certainly room for improvement, especially regarding a more tailored UI for Clickhouse… we plan to continue exploring potential solutions.”

Here’s a distilled version of the common pain points we experienced:

• Ad-hoc Querying Was Painful: Writing raw Clickhouse SQL in Metabase for quick log searches felt cumbersome and slow. Even modifying existing complex query templates was error-prone – a tiny syntax mistake could lead to minutes spent debugging the query itself, especially stressful during production incidents.
• Disconnect Between Visualization and Raw Logs: A common workflow is to visualize trends (e.g., errors over time) and then drill down into the specific logs causing those trends. In Metabase, this often meant writing two separate queries – one for aggregation/visualization and another (often rewritten from scratch) just to see the raw log lines. Metabase’s row limits (around 2k) further complicated viewing the full context of raw logs after filtering. The intuitive “slice and drill-down” experience many log tools offer was missing.
• UI/UX Annoyances: Several smaller but cumulative issues added friction: difficulty selecting precise time ranges like “last 6 hours,” viewing logs immediately surrounding a relevant event, columns getting truncated (...), and limited timestamp precision display in results. Though there are some workarounds, they often felt like band-aids rather than solutions.

TL;DR: Metabase interface wasn’t optimized for the specific task of log exploration. Debugging sessions that should have taken minutes were stretching significantly longer. Querying and exploring logs felt clunkier than it needed to be.

And one fine day, I decided to stop just wishing for a better tool and start building one:

Logchef#

When I first started prototyping, I kept the scope pretty tight: just build a viewer for the standard OTEL schema. OTEL’s flexible enough, but a quick chat with Kailash sparked what turned out to be a game-changing idea: make Logchef schema-agnostic. And that really became the core concept.

Basically, Logchef lets you connect it straight to your existing Clickhouse log tables, no matter their structure. All it really needs is a timestamp field (DateTime or DateTime64). Bring your own custom schemas, stick with the OTEL standard, or even adapt it to your own needs. Logchef doesn’t force you into a specific format. From what I’ve seen, not many tools offer this kind of plug-and-play flexibility with existing tables today.

Logchef is designed as a specialized query and visualization layer sitting on top of Clickhouse. Logchef intentionally excludes log collection and ingestion. Why reinvent the wheel when excellent tools like Vector, Fluentbit, Filebeat, etc., already handle this reliably? Logchef focuses purely on exploring the logs once it’s in Clickhouse.

Stack#

• Backend: Written in Go for performance and concurrency.
• Metadata Storage: Uses SQLite for lightweight management of users, teams, Clickhouse source connections, and query collections. It’s simple and perfectly suited for this kind of a metadata store.
• Frontend: An interactive log viewer with Vue.js and styled with shadcn/ui and Tailwind CSS. I also implemented a simple search syntax for common filtering tasks (e.g., status=200 and path~"/api/"). This involved writing a tokenizer and parser that translates this syntax into efficient ClickHouse SQL conditions optimised for querying logs. Building this parser, validator, and integrating it smoothly with the Monaco editor for syntax highlighting was a significant effort but quite happy with the end result.

Setting Up the Public Demo (demo.logchef.app)#

I wanted a public demo instance so people could try Logchef easily. Setting this up involved a few specific tweaks compared to a standard deployment, all managed within the Docker Compose setup:

1. Generating Dummy Data: A log viewer isn’t much use without logs! Instead of ingesting real data, I configured vector using its demo_logs source type. This continuously generates realistic-looking syslog and HTTP access logs and pushes them into the demo Clickhouse instance (syslogs and http_logs tables). It gives users immediate data to query without any setup on their part.

   # vector.toml snippet
   [sources.generate_syslog]
   type = "demo_logs"
   format = "syslog"
   interval = 0.3 # Generate logs frequently
   
   [sinks.clickhouse_syslog]
   # ... config to send to Clickhouse ...
   table = "syslogs"

2. Securing Admin Endpoints (Demo Mode): Since this is a public, shared instance, I wanted to prevent users from making potentially disruptive changes via the API (like deleting sources or teams). I used Caddy as the reverse proxy and configured it to intercept requests to admin-specific API routes (like /api/v1/admin/*) and block any method other than GET. If someone tries a POST, PUT, or DELETE to these endpoints, Caddy returns a 403 Forbidden directly. This keeps the demo environment stable.

   Caddyfile snippet (conceptual)
   handle /api/v1/admin/* {
       @block_methods method POST PUT DELETE PATCH
       respond @block_methods `{"error":"Operation not permitted in demo mode"}` 403
       reverse_proxy logchef:8125 # Forward GET requests
   }

3. Improving Demo Login UX: Logchef uses OIDC for authentication. For the demo, I’m running Dex as the OIDC provider. To make it completely frictionless for users, I didn’t want them needing to sign up or guess credentials. I simply customized Dex’s theme template for the login page to explicitly display the static demo username (demo@logchef.app) and password (password) right there. It’s a small UX tweak (again, thanks to Kailash for the idea!), but it means anyone landing on the demo can log in instantly.

   <!-- Dex login template snippet -->
   <div class="dex-info-box">
     <strong>Demo Credentials:</strong><br>
     Username: <code>demo@logchef.app</code><br>
     Password: <code>password</code>
   </div>
   <input ... value="demo@logchef.app" .../>
   <input ... type="password" value="password" .../>

What’s Next?#

Logchef is already being used internally, but the journey towards a full v1.0 release continues this year. The roadmap includes exciting additions like:

• Alerting: Trigger notifications based on query results.
• Live Tail Logs: Stream logs in real-time.
• Enhanced Dashboarding: More powerful visualization capabilities.

Logchef is open source (AGPLv3), and community involvement is welcomed. You can check out the Demo or view the code on GitHub.

If you have more ideas or features you’d like to see, please reach out on GitHub or email me! I’m always open to suggestions and feedback.

Honestly, building Logchef has been incredibly rewarding. It started as a way to fix something that bugged me (and others!), and seeing it turn into a tool I’m genuinely excited about feels great.

I couldn’t have done it alone, though. I’m really grateful to my friends and colleagues who jumped in with feedback along the way. Huge thanks to Kailash for the constant support and encouragement, and to Vivek, Sarat, and Rohan for testing the early builds and offering great suggestions.

Finally, a big thank you to my wife, who patiently endured my late-night coding sessions. Her support means the world to me <3

Fin!

]]> Karan Sharma https://workdone0.substack.com/p/motorcycle-engines https://workdone0.substack.com/p/motorcycle-engines Sun, 27 Apr 2025 00:00:00 GMT Shubham Kumar https://workdone0.substack.com/p/cost-of-lies https://workdone0.substack.com/p/cost-of-lies Wed, 23 Apr 2025 00:00:00 GMT Shubham Kumar https://workdone0.substack.com/p/feynman-letter https://workdone0.substack.com/p/feynman-letter Tue, 22 Apr 2025 00:00:00 GMT Shubham Kumar https://workdone0.substack.com/p/deep-learning-intro https://workdone0.substack.com/p/deep-learning-intro Tue, 15 Apr 2025 00:00:00 GMT Shubham Kumar https://www.evalapply.org/posts/demo-clojure-workflow-scicloj/index.html https://www.evalapply.org/posts/demo-clojure-workflow-scicloj/index.html Fri, 28 Mar 2025 00:00:00 GMT Aditya Athalye meta riff howto whyto clojure tools_for_thought programming ai intelligence_augmentation https://mrkaran.dev/posts/trying-nixos/ https://mrkaran.dev/posts/trying-nixos/ Sun, 23 Mar 2025 04:45:00 GMT " "" "" ]; # Optimize fetching from GitHub connect-timeout = 5; # Prevent unneeded rebuilds commit-lockfile-summary = "Update flake.lock"; }; # Garbage collection settings gc = { automatic = true; dates = "weekly"; options = "--delete-older-than 30d"; }; # Optimize builds using different build cores buildCores = 0; # 0 means use all available cores # Enable flakes and modern Nix command features extraOptions = '' experimental-features = nix-command flakes warn-dirty = false keep-going = true log-lines = 20 ''; }; Escape Hatches# So far things seems all rosy. Within just spending a couple of minutes - I had a perfectly working machine for myself - and the best part - all reproducible with a single command. I was starting to see why people who use NixOS preach about it so much. However, not everything is smooth when you deviate from the happy path. For instance, I use Aider for LLM assisted programming, but the version on Nixpkgs was about three minor versions behind. Typically for any other software, I wouldn’t have cared so much - however with these LLM tools, a lot changes rapidly and I didn’t want to stay behind. Besides, it seemed like a fun exercise in getting my hands dirty by installing a Python package on NixOS which turned out to be quite tricky because Nix is absurdly obsessive about fully isolated builds. Here’s an example flake that I used for attempting to install Aider with uv in a dev shell (which didn’t work btw): { description = "Aider development environment"; inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; flake-utils.url = "github:numtide/flake-utils"; }; outputs = { self, nixpkgs, flake-utils }: flake-utils.lib.eachDefaultSystem (system: let pkgs = nixpkgs.legacyPackages.${system}; in { devShell = pkgs.mkShell { buildInputs = with pkgs; [ python312 uv ]; shellHook = '' export PATH="$HOME/.local/bin:$PATH" ''; }; } ); } Entering the dev shell with nix develop and installing Aider with uv: uv tool install --force --python python3.12 aider-chat@latest However, I ran into this error: "/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/litellm/litellm_core_uti ls/llm_cost_calc/utils.py", line 9, in from litellm.utils import get_model_info File "/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/litellm/utils.py", line 53, in from tokenizers import Tokenizer File "/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/tokenizers/__init__.py", line 78, in from .tokenizers import ( ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory The error indicated that Aider was missing a required dependency libstdc++.so.6 which is a part of the C++ standard library needed by the tokenizers package. To fix this, I added stdenv.cc.cc.lib (and even gcc to be on the safer side) to my buildInputs. This is because while uv installs Python packages, it doesn’t handle system-level dependencies. In a Nix environment, every dependency, including system libraries, must be explicitly specified. Frankly, Python’s packaging ecosystem is still a mess. Although tools like uv help, achieving a completely isolated build, especially when shared libraries are involved is challenging. I wish the Python community would put more effort into resolving these issues. While I was able to make aider work by explicitly adding all the dependencies, I faced another outdate package: code-cursor. Since this is a full blown electron app, I didn’t wish to package this myself. After some frustration, I tried using Distrobox as recommended by a colleague. Distrobox lets you run containers that feel almost like a native OS by managing user IDs, host mounts, network interfaces, and more. I used an Arch Linux image, installed cursor-bin from the AUR, and everything worked fine. Well mostly: Fonts were missing. So, if I want to use custom fonts in my IDE - I need to have them installed in the container as well. Since my fish shell config had export EDITOR=nvim, I had to install neovim in the container as well, otherwise, I’d get an error when trying to git commit etc. There’s an option to customise the shell in distrobox, but for whatever reason (that I didn’t want to debug), it didn’t work for me. Yet, something still felt off. The whole point of using NixOS is to achieve a fully declarative and reproducible setup. Resorting to an escape hatch like Distrobox undermines that goal. So I was very conflicted about this. I’m sure there’s a better way to handle these situations, and I should probably read the docs to find out. Final Thoughts# I’m definitely sold on running NixOS, especially when managing multiple systems. With a single declarative file (configuration.nix), duplicating your setup across machines becomes effortless. No more “documenting” (or rather forgetting to document and keeping it updated)- as the config is the single source of truth. Fun fact: I even messed up my NixOS build by misconfiguring the hardware-configuration.nix, and my system became unusable even after a reboot, it couldn’t mount the filesystem on the correct device. In other distros, that would have sent me into panic mode, but with NixOS, all I had to do was revert to the previous working state, and everything was fine. That was so cool! I’m definitely considering moving my homelab to NixOS in the coming few days because I honestly see the value for a server setup. I often set up my personal server and then forget everything I’ve done and I’m always scared of touching or creating a new server from scratch. I even created a small shell script installer to help me for getting a base system ready. But like this shell script or even tools such as Ansible - they are all idempotent in nature. However in Nix, if I remove a certain piece from the configuration, there isn’t a trace of it left on the system. That makes it truly declarative and reproducible - unlike Ansible where you can still have some parts of the old setup. However, for my primary machine at work, I’ll wait on the sidelines until the packages I depend on resolve their dependency issues and I get a chance to read up more on the escape hatches I tried to see if there’s a more streamlined way of doing things. I might be missing a lot of fundamental details since I skipped the docs entirely to get my hands dirty. But now that I see the value of a declarative system and especially how easy it is to roll back the machine to a previously known good state, I’m motivated to read up more on this and might post an update to this blog. Fin!]]> I’ve been introduced to Nix by my colleagues at work. Being a Linux user for over a decade and a serial distro hopper, I was curious to learn more about it. I’d seen Nix mentioned before, but the comments about its steep learning curve made me wonder if the effort was worth it. I decided to give it a try by reading this excellent beginner’s guide however got bored very quickly and decided to “learn on the fly”. I spun up a VM in my homelab to install NixOS using their official GUI installer image.

Installation & First Impressions#

The installation was as straightforward as any other Linux distro. NixOS is a declarative operating system that leverages the Nix functional package manager and a rich ecosystem of Nix packages. The flexibility is mind-blowing: you can configure everything—from user accounts and SSH keys to $SHELL config and plugins entirely through code.

Once installed, the first place you’d want to poke around is the /etc/nixos directory, which contains two essential configuration files:

• hardware-configuration.nix: Generated during installation (or regenerated with commands like nixos-generate-config), it has hardware-specific details such as filesystem mount points, disk configurations, kernel modules etc. See an example file here.

• configuration.nix: This is the most important file you want to start editing with. Here you define system-wide settings like timezone, locale, user accounts, and networking. Everything is declared in one place, making your system’s state reproducible.

First Configuration Changes#

When I opened the terminal, I immediately noticed that vim wasn’t installed. So, I updated my configuration.nix to include the packages I needed:

environment.systemPackages = with pkgs; [
  git
  vim
];

After saving, I ran:

sudo nixos-rebuild switch

This rebuilds the system using the new declarative configuration.

Version Control & Flakes#

Next, I wanted to set up version control for my Nix configurations. The key takeaway is that while the system’s state is revertable in NixOS, your personal data (which includes configuration.nix) isn’t automatically backed up. You must manage your own version history for your Nix configs. Since I was tweaking with no knowledge of Nix, having a version history was crucial.

I moved my /etc/nixos configs to ~/Code/nixos-configs and initialized a Git repository:

# Create repo in home directory (better than root-owned /etc/nixos)
mkdir ~/nixos-config
cp -r /etc/nixos/* ~/nixos-config/
cd ~/nixos-config

# Initialize Git
git init
git add .
git commit -m "Initial NixOS configuration"

# Add GitHub remote
git remote add origin https://github.com/username/nixos-config.git
git push -u origin main

Here’s how flake.nix looks:

{
  description = "NixOS configuration for Karan's homelab, servers, and personal dev machines";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    
    # Add agenix as an input
    agenix = {
      url = "github:ryantm/agenix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    # Optionally add other inputs like home-manager
    # home-manager = {
    #   url = "github:nix-community/home-manager/release-24.11";
    #   inputs.nixpkgs.follows = "nixpkgs";
    # };
  };

  outputs = { self, agenix, nixpkgs, ... }@inputs: {
    # Make agenix available as a package
    packages.x86_64-linux.agenix = agenix.packages.x86_64-linux.default;
    
    nixosConfigurations.work = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        ./configuration.nix
        agenix.nixosModules.default  # Add agenix module
      ];
    };
  };
}

A Note on Flakes#

Flakes are an experimental (although widely adopted in the community) feature in Nix that bring reproducibility, composability, and a standardized structure to your configurations and package definitions. They allow you to declare all inputs (like nixpkgs, home-manager, or other repositories) and outputs (such as system configurations, packages, or development shells) in a single file. Flakes also create a lock file (flake.lock) that pins your dependencies to specific revisions, ensuring that your builds remain reproducible over time.

I learned the hard way that—even for local configurations you must commit your files. Otherwise, you may see errors like:

path '/nix/store/...source/flake.nix' does not exist

Even if you’re using local paths and have no intention to push to git, you still need git init && git add for flakes to work. From whatever google-fu I did, it seems this requirement is to ensure that flakes can reliably reference the exact content in your configuration. I am sure there might be good reasons for it (as I said before, I’ve skipped RTFMing altogether ^_^), but atleast the errors can be more verbose/helpful.

And why I skipped docs: Remember, we’re on a mission to get things up and running with Nix and then later spend time about reading their internals if it actually proves to be a valuable experiment.

Switching Channels#

While installing packages, I noticed some packages were quite outdated. That’s when I learned about NixOS channels. Think of channels as analogous to LTS releases. For faster updates, you can switch to the unstable channel. Although the name sounds intimidating, it simply means you’ll receive more frequent package updates.

To do this, you can edit your flake.nix and switch the URL to an unstable channel:

  inputs = {
    - nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11"; # Stable channel
    + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; # Unstable channel
  };

Firmware Updates#

After setting up packages, it was time to configure firmware updates using fwupd—essential for keeping your hardware up to date.

I asked Claude to help me for a quick setup. Here’s what I did:

{ config, pkgs, ... }:

{
  services.fwupd.enable = true;
}

Then run a rebuild:

sudo nixos-rebuild switch

Once enabled, you can use the fwupdmgr command-line tool to manage firmware updates:

# Refresh metadata and check for available updates
fwupdmgr refresh
fwupdmgr get-updates
# Install available firmware updates
fwupdmgr update

Fine Tuning#

I also tweaked some settings for the Nix package manager to optimize builds, caching, and overall performance. Here’s a snippet from my configuration:

  # Nix package manager optimizations
  nix = {
    settings = {
      # Optimize store to remove duplicate files
      auto-optimise-store = true;

      # Allow building multiple derivations in parallel
      max-jobs = "auto";

      # Number of parallel build tasks per job
      cores = 0; # 0 means use all available cores

      # Use the binary cache aggressively
      substituters = [
        "<https://cache.nixos.org>"
        "<https://nix-community.cachix.org>"
        "<https://nixpkgs-wayland.cachix.org>"
      ];

      # Optimize fetching from GitHub
      connect-timeout = 5;

      # Prevent unneeded rebuilds
      commit-lockfile-summary = "Update flake.lock";
    };

    # Garbage collection settings
    gc = {
      automatic = true;
      dates = "weekly";
      options = "--delete-older-than 30d";
    };

    # Optimize builds using different build cores
    buildCores = 0; # 0 means use all available cores

    # Enable flakes and modern Nix command features
    extraOptions = ''
      experimental-features = nix-command flakes
      warn-dirty = false
      keep-going = true
      log-lines = 20
    '';
  };

Escape Hatches#

So far things seems all rosy. Within just spending a couple of minutes - I had a perfectly working machine for myself - and the best part - all reproducible with a single command. I was starting to see why people who use NixOS preach about it so much.

However, not everything is smooth when you deviate from the happy path. For instance, I use Aider for LLM assisted programming, but the version on Nixpkgs was about three minor versions behind. Typically for any other software, I wouldn’t have cared so much - however with these LLM tools, a lot changes rapidly and I didn’t want to stay behind. Besides, it seemed like a fun exercise in getting my hands dirty by installing a Python package on NixOS which turned out to be quite tricky because Nix is absurdly obsessive about fully isolated builds.

Here’s an example flake that I used for attempting to install Aider with uv in a dev shell (which didn’t work btw):

{
  description = "Aider development environment";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        devShell = pkgs.mkShell {
          buildInputs = with pkgs; [
            python312
            uv
          ];
          shellHook = ''
            export PATH="$HOME/.local/bin:$PATH"
          '';
        };
      }
    );
}

Entering the dev shell with nix develop and installing Aider with uv:

uv tool install --force --python python3.12 aider-chat@latest

However, I ran into this error:

"/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/litellm/litellm_core_uti
ls/llm_cost_calc/utils.py", line 9, in <module>
    from litellm.utils import get_model_info
  File
"/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/litellm/utils.py", line
53, in <module>
    from tokenizers import Tokenizer
  File
"/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/tokenizers/__init__.py",
line 78, in <module>
    from .tokenizers import (
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

The error indicated that Aider was missing a required dependency libstdc++.so.6 which is a part of the C++ standard library needed by the tokenizers package. To fix this, I added stdenv.cc.cc.lib (and even gcc to be on the safer side) to my buildInputs. This is because while uv installs Python packages, it doesn’t handle system-level dependencies. In a Nix environment, every dependency, including system libraries, must be explicitly specified.

Frankly, Python’s packaging ecosystem is still a mess. Although tools like uv help, achieving a completely isolated build, especially when shared libraries are involved is challenging. I wish the Python community would put more effort into resolving these issues.

While I was able to make aider work by explicitly adding all the dependencies, I faced another outdate package: code-cursor. Since this is a full blown electron app, I didn’t wish to package this myself.

After some frustration, I tried using Distrobox as recommended by a colleague. Distrobox lets you run containers that feel almost like a native OS by managing user IDs, host mounts, network interfaces, and more. I used an Arch Linux image, installed cursor-bin from the AUR, and everything worked fine. Well mostly:

• Fonts were missing. So, if I want to use custom fonts in my IDE - I need to have them installed in the container as well.
• Since my fish shell config had export EDITOR=nvim, I had to install neovim in the container as well, otherwise, I’d get an error when trying to git commit etc. There’s an option to customise the shell in distrobox, but for whatever reason (that I didn’t want to debug), it didn’t work for me.

Yet, something still felt off. The whole point of using NixOS is to achieve a fully declarative and reproducible setup. Resorting to an escape hatch like Distrobox undermines that goal. So I was very conflicted about this. I’m sure there’s a better way to handle these situations, and I should probably read the docs to find out.

Final Thoughts#

I’m definitely sold on running NixOS, especially when managing multiple systems. With a single declarative file (configuration.nix), duplicating your setup across machines becomes effortless. No more “documenting” (or rather forgetting to document and keeping it updated)- as the config is the single source of truth.

Fun fact: I even messed up my NixOS build by misconfiguring the hardware-configuration.nix, and my system became unusable even after a reboot, it couldn’t mount the filesystem on the correct device. In other distros, that would have sent me into panic mode, but with NixOS, all I had to do was revert to the previous working state, and everything was fine. That was so cool!

I’m definitely considering moving my homelab to NixOS in the coming few days because I honestly see the value for a server setup. I often set up my personal server and then forget everything I’ve done and I’m always scared of touching or creating a new server from scratch. I even created a small shell script installer to help me for getting a base system ready. But like this shell script or even tools such as Ansible - they are all idempotent in nature. However in Nix, if I remove a certain piece from the configuration, there isn’t a trace of it left on the system. That makes it truly declarative and reproducible - unlike Ansible where you can still have some parts of the old setup.

However, for my primary machine at work, I’ll wait on the sidelines until the packages I depend on resolve their dependency issues and I get a chance to read up more on the escape hatches I tried to see if there’s a more streamlined way of doing things. I might be missing a lot of fundamental details since I skipped the docs entirely to get my hands dirty. But now that I see the value of a declarative system and especially how easy it is to roll back the machine to a previously known good state, I’m motivated to read up more on this and might post an update to this blog.

Fin!

]]> Karan Sharma https://workdone0.substack.com/p/python_decorator https://workdone0.substack.com/p/python_decorator Sat, 22 Mar 2025 00:00:00 GMT Shubham Kumar https://ravidwivedi.in/posts/libreoffice-conference-2024/ https://ravidwivedi.in/posts/libreoffice-conference-2024/ Fri, 14 Mar 2025 16:18:39 GMT Last year, I attended the annual LibreOffice Conference in Luxembourg with the help of a generous travel grant by The Document Foundation (TDF). It was a three-day event from the 10th to the 12th of October 2024, with an additional day for community meetup on the 9th.

Luxembourg is a small country in Western Europe. It is insanely wealthy with high living standards. After going through an arduous visa process, I got to the country on the 8th of October. Upon arriving in Luxembourg, I took a bus to the city center, where my hotel — Park Inn — was located. I deboarded the bus at the Luxembourg Central station. Before walking towards my hotel, I stopped to click a few pictures of the beautiful station.

All the public transport in Luxembourg was free of cost. The experience of being in Luxembourg was as if I had stepped in another world. The roads had separate tracks for cycling and separate lanes for buses, along with wide footpaths. In addition, the streets were pretty neat and clean.

The conference venue was in Belval, while I stayed in the city center. Even though my stay was 20 km from the conference venue, the commute was convenient thanks to free of cost train connections. The train rides were comfortable, smooth, and scenic, covering the distance in half an hour. Moreover, I never found the trains to be very crowded, which enabled me to always get a seat.

My breakfast was included in the hotel booking. The breakfast had many options. It had coffee and fruit juices, along with diverse food options. Some of the items I remember were croissant, pain au chocolat, brie (a type of cheese), scrambled eggs, boiled eggs, and various types of meat dishes. Other than this, there were fruits such as pears.

Pre-conference, a day was reserved for the community meetup on the 9th of October. On that day, the community members introduced themselves and their contributions to the LibreOffice project. It acted as a brainstorming session. All the attendees got a lovely conference bag, which contained a T-Shirt, a pen and a few stickers. I also met my long time collaborators Mike, Sophie and Italo from the TDF, whom I had interacted only remotely till then. Likewise, I also met TDF’s sysadmin Guilhem, who I interacted before regarding setting up my LibreOffice mirror.

The conference started on the 10th. There were 5 attendees from India, including me, while most of the attendees were from Europe. The talks were in English. One of the talks that stood out for me was about Luxchat — a chat service run by the Luxembourg government based on the Matrix protocol for the citizens of Luxembourg. I also liked Italo’s talk on why document formats must be freedom-respecting. On the first night, the conference took us to a nice dinner in a restaurant. It offered one more way to socialize with other attendees and explore food at the same time.

On the 11th of October, I went for a walk in the morning with Biswadeep for some sightseeing around our hotel area. As a consequence, I missed the group photo of the conference, which I wanted to be in. Anyway, we enjoyed roaming around the picturesque Luxembourg city. We also sampled a tram ride to return to our hotel.

The conference ended on the 12th with a couple of talks. This conference gave me an opportunity to meet the global LibreOffice community, connect and share ideas. It also gave me a peek into the country of Luxembourg and its people, where I had good experience. English was widely known, and I had no issues getting by.

Thanks to all the organizers and sponsors of the conference!

]]> Ravi Dwivedi https://www.divyamohan.com/riscing-it/ https://www.divyamohan.com/riscing-it/ Thu, 06 Mar 2025 14:27:33 GMT

A couple of days back, Reuters reported China's plans to issue guidance encouraging the use of open source RISC-V chips nationwide, in a bid to reduce dependence on western-owned technologies. Given the state of affairs today, this could have significant implications on the global open source and semiconductor ecosystems.

What is RISC-V?

Pronounced as Risk-five, RISC-V is an open standard instruction set architecture based on Reduced Instruction Set Architecture (RISC) principles. Owing to its open source nature, it's deemed an attractive option in comparison with more proprietary alternatives that dominate the ISA space.

China's RISC-V stance

According to Reuters, China's decision to promote RISC-V nationwide is part of a broader strategy to enhance its semiconductor self-reliance and promote technological autonomy. The policy, expected to be released soon, will be drafted by several key government bodies, including the Ministry of Industry and Information Technology and the Cyberspace Administration of China. This move reflects China's commitment to leveraging RISC-V as a geopolitically neutral technology to reduce its reliance on US-dominated semiconductor standards.

Impact on the global open source ecosystem

While its open source nature encourages a collaborative environment contributing to the evolution of the ISA & related extensions, as well as promoting adoption, existing geopolitical tensions between the West & China could lead to a RISC-V ecosystem that is fractured at its core.

This is because, as explored in this New Stack Article by Amanda Brock earlier this year, laws trump licensing. Since open source is subject to the laws of the land, these tensions (potential sanctions and/or tariffs) could also lead to different countries developing and maintaining their versions of RISC-V. Not only would this impede interoperability and future adoption, but it would also affect further upstream development reliant on a diverse global community, complicating standardization and maturity.

In conclusion, China's upcoming RISC-V policy will have profound implications for the global tech ecosystem, influencing both the open source community and the broader semiconductor industry supply chain. While presenting opportunities for greater diversity and collaboration, there's also the looming threat of a fragmented ecosystem and delayed maturity, due to existing global tensions.

]]> Divya Mohan policy tech Open Source https://mrkaran.dev/posts/playo-badminton/ https://mrkaran.dev/posts/playo-badminton/ Mon, 03 Mar 2025 18:40:55 GMT =3.12" # dependencies = [ # "click", # "requests", # "pytz", # "rich", # "python-dateutil", # "python-telegram-bot", # ] # /// import click import requests import json import datetime import pytz import os import sys from rich.console import Console from rich.table import Table from dateutil import parser from telegram import Bot, InputMediaPhoto from telegram.constants import ParseMode from io import BytesIO import asyncio console = Console() @click.command() @click.option("--lat", default=12.9783692, help="Latitude for search") @click.option("--lng", default=77.6408356, help="Longitude for search") @click.option("--radius", default=50, help="City radius in km") @click.option("--sport", default="SP5", help="Sport ID (default: SP5 for Badminton)") @click.option("--start-time", default="19:00", help="Desired start time (24-hour format HH:MM)") @click.option("--end-time", default="20:00", help="Desired end time (24-hour format HH:MM)") @click.option("--timezone", default="Asia/Kolkata", help="Your timezone") @click.option("--verbose", is_flag=True, help="Show detailed information including exact UTC/IST times") @click.option("--include-full", is_flag=True, help="Include games that are full") @click.option("--telegram", is_flag=True, help="Send results to Telegram") @click.option("--telegram-token", envvar="TELEGRAM_BOT_TOKEN", help="Telegram Bot Token (or set TELEGRAM_BOT_TOKEN env var)") @click.option("--telegram-chat-id", envvar="TELEGRAM_CHAT_ID", help="Telegram Chat ID (or set TELEGRAM_CHAT_ID env var)") def find_games(lat, lng, radius, sport, start_time, end_time, timezone, verbose, include_full, telegram, telegram_token, telegram_chat_id): """Find available badminton games on Playo matching your criteria.""" # Get today's date in the specified timezone local_tz = pytz.timezone(timezone) now = datetime.datetime.now(local_tz) today_date = now.strftime("%Y-%m-%dT%H:%M:%S.%fZ") # Parse desired time window try: desired_start = datetime.datetime.strptime(start_time, "%H:%M").time() desired_end = datetime.datetime.strptime(end_time, "%H:%M").time() except ValueError: console.print("[bold red]Error:[/bold red] Invalid time format. Please use HH:MM (24-hour format).") return console.print(f"[bold green]Searching for badminton games around your location...[/bold green]") console.print(f"Looking for games between [bold]{start_time}[/bold] and [bold]{end_time}[/bold] IST today") if verbose: console.print(f"[dim]Search parameters: lat={lat}, lng={lng}, radius={radius}km[/dim]") console.print(f"[dim]Current time in {timezone}: {now.strftime('%Y-%m-%d %H:%M:%S')}[/dim]") # Prepare API request url = "https://api.playo.io/activity-public/list/location" payload = { "lat": lat, "lng": lng, "cityRadius": radius, "gameTimeActivities": False, "page": 0, "lastId": "", "sportId": [sport], "booking": False, "date": [today_date] } headers = { "Content-Type": "application/json", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" } try: response = requests.post(url, headers=headers, json=payload) response.raise_for_status() data = response.json() if data.get("requestStatus") != 1 or "data" not in data: console.print("[bold red]Error:[/bold red] Failed to get valid response from Playo API") return # Process activities activities = data["data"].get("activities", []) if not activities: console.print("[yellow]No badminton activities found for today[/yellow]") return # Filter activities based on criteria matching_games = [] for activity in activities: # Convert UTC times to local timezone start_time_utc = parser.parse(activity["startTime"]) end_time_utc = parser.parse(activity["endTime"]) start_time_local = start_time_utc.astimezone(local_tz) end_time_local = end_time_utc.astimezone(local_tz) # Print all times in debug mode # console.print(f"DEBUG: {activity.get('location', 'Unknown')} - Start: {start_time_local.strftime('%H:%M')} IST (UTC: {start_time_utc.strftime('%H:%M')})") # Convert time objects correctly for comparison start_hour = start_time_local.hour start_minute = start_time_local.minute # Convert desired times to hours and minutes for easier comparison desired_start_hour = desired_start.hour desired_start_minute = desired_start.minute desired_end_hour = desired_end.hour desired_end_minute = desired_end.minute # Check if this game starts at 7PM (19:00) and ends at 8PM (20:00) is_time_match = False # Get duration in minutes duration_minutes = ((end_time_local.hour * 60 + end_time_local.minute) - (start_time_local.hour * 60 + start_time_local.minute)) # Check if start time is 7PM (with small tolerance) if (start_hour == desired_start_hour and start_minute >= desired_start_minute and start_minute < desired_start_minute + 10): # Allow a small window of 10 minutes # Check if duration is approximately 1 hour (between 50-70 minutes) if 50 <= duration_minutes <= 70: is_time_match = True # Check if there are available slots is_available = ( not activity.get("full", True) and (activity.get("maxPlayers", 0) == -1 or activity.get("joineeCount", 0) < activity.get("maxPlayers", 0)) ) # When verbose, print time details for each game to help debug if verbose: time_info = f"[dim]{activity.get('location', 'Unknown')} - Start: {start_time_local.strftime('%H:%M')} IST ({start_time_utc.strftime('%H:%M')} UTC), " + \ f"End: {end_time_local.strftime('%H:%M')} IST, Duration: {duration_minutes} min, " + \ f"Time match: {'Yes' if is_time_match else 'No'}, Available: {'Yes' if is_available else 'No'}[/dim]" console.print(time_info) # Both conditions must be true if is_time_match and is_available: matching_games.append({ "id": activity["id"], "location": activity["location"], "venue_name": activity.get("venueName", "N/A"), "start": start_time_local.strftime("%I:%M %p"), "end": end_time_local.strftime("%I:%M %p"), "players": f"{activity.get('joineeCount', 0)}/{activity.get('maxPlayers', 'unlimited')}", "host": activity.get("userInfo", [{}])[0].get("fName", "Unknown"), "skill": activity.get("skill", "Any"), "price": f"{activity.get('price', 0)} {activity.get('currencyTxt', 'INR')}" }) # Display results if matching_games: table = Table(title=f"Available Badminton Games ({len(matching_games)} matches found)") table.add_column("Location", style="cyan") table.add_column("Time", style="green") table.add_column("Players", style="yellow") table.add_column("Host", style="magenta") table.add_column("Skill Level", style="blue") table.add_column("Link", style="bright_blue") for game in matching_games: table.add_row( f"{game['venue_name']}", f"{game['start']} - {game['end']}", game["players"], game["host"], game["skill"], f"https://playo.co/match/{game['id']}" ) console.print(table) # Send to Telegram if requested if telegram: if not telegram_token or not telegram_chat_id: console.print("[bold red]Error:[/bold red] Telegram token and chat ID are required for Telegram notifications") console.print("[dim]Set them with --telegram-token and --telegram-chat-id or via environment variables[/dim]") else: try: send_to_telegram(matching_games, telegram_token, telegram_chat_id) console.print("[green]Results sent to Telegram successfully![/green]") except Exception as e: console.print(f"[bold red]Error sending to Telegram:[/bold red] {e}") else: console.print("[yellow]No games found matching your criteria[/yellow]") if telegram and telegram_token and telegram_chat_id: try: asyncio.run(send_telegram_message( "No badminton games found matching your criteria for today.", telegram_token, telegram_chat_id )) console.print("[green]Empty results notification sent to Telegram[/green]") except Exception as e: console.print(f"[bold red]Error sending to Telegram:[/bold red] {e}") except requests.RequestException as e: console.print(f"[bold red]Error:[/bold red] Failed to connect to Playo API: {e}") except json.JSONDecodeError: console.print("[bold red]Error:[/bold red] Failed to parse API response") except Exception as e: console.print(f"[bold red]Error:[/bold red] An unexpected error occurred: {e}") def send_to_telegram(games, token, chat_id): """Send game information to Telegram as a nicely formatted message.""" if not games: return # Create a formatted message for Telegram message = "🏸 *Available Badminton Games* 🏸\n\n" for i, game in enumerate(games, 1): message += f"*{i}. {game['venue_name']}*\n" message += f"⏰ {game['start']} - {game['end']}\n" message += f"👥 Players: {game['players']}\n" message += f"👤 Host: {game['host']}\n" message += f"🎯 Skill: {game['skill']}\n" message += f"🔗 [Join Game](https://playo.co/match/{game['id']})\n\n" # Send the message asyncio.run(send_telegram_message(message, token, chat_id)) async def send_telegram_message(message, token, chat_id): """Send a message to Telegram using the Bot API.""" bot = Bot(token=token) await bot.send_message( chat_id=chat_id, text=message, parse_mode=ParseMode.MARKDOWN, disable_web_page_preview=False ) if __name__ == "__main__": find_games() The script outputs a beautiful output: Telegram: Scheduling# I wanted this script to run reliably every day and used GitHub Actions for that. GitHub Actions felt like the path of least resistance as I didn’t have to worry about keeping a server running or getting alerts if something crashed. For a small personal script like this, it was the perfect “set it and forget it” solution. name: Badminton Game Checker Base on: schedule: # Run Monday to Friday at 12:00 PM IST (6:30 AM UTC) - cron: "30 6 * * 1-5" jobs: check-games: runs-on: ubuntu-latest env: TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }} TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }} LATITUDE: ${{ inputs.latitude }} LONGITUDE: ${{ inputs.longitude }} RADIUS: ${{ inputs.radius }} SPORT_ID: ${{ inputs.sport_id }} TIMEZONE: ${{ inputs.timezone }} START_TIME: ${{ inputs.start_time }} END_TIME: ${{ inputs.end_time }} steps: - name: Checkout code uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" cache: "pip" - name: Install dependencies run: | python -m pip install --upgrade pip pip install uv - name: Run game check run: | echo "Checking for games from $START_TIME to $END_TIME" uv run finder.py \ --lat "$LATITUDE" \ --lng "$LONGITUDE" \ --radius "$RADIUS" \ --sport "$SPORT_ID" \ --timezone "$TIMEZONE" \ --start-time "$START_TIME" \ --end-time "$END_TIME" \ --telegram env: TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }} TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }} I used GitHub Actions inputs to configure the variables for my script. Found this feature to be quite neat for scheduling different crons for weekday/weekends. Summary# For small quality-of-life improvements - solving your own specific problems with custom scripts tailored exactly to your needs - gotta love the LLMs man. We’re gonna see more and more of such “personal tooling” in future as the entry to barrier for coding is lowered with LLMs. The democratization of coding through LLMs means people (even non-technical ones) can focus on “describing” the problem well, rather than struggling with implementation details. Being able to articulate what you want clearly becomes the primary skill - yes, it’s a skill issue if you can’t prompt well, but it’s far more accessible than learning programming from scratch. Fin!]]> I’ve been playing badminton more regularly since the start of 2025 - almost 4-5 days a week. I recently moved to a new part of the city, which meant I couldn’t play with my old friends anymore. PlayO has been super helpful for finding games with new people. On PlayO, a host creates a game and up to 6 people can join one court for a one-hour badminton doubles session.

However, on hectic days I would often forget to check for badminton games, only to find them fully booked later. I wanted to automate this process by creating a small script that would send me scheduled alerts about today’s game availability, allowing me to book slots before they filled up. I drew inspiration from Matt’s post where he did something similar.

Thankfully, PlayO has a public API endpoint to retrieve a list of available games: https://api.playo.io/activity-public/list/location.

You can send a POST request to this URL with these parameters for filtering:

{
  "lat": 12.9783692,
  "lng": 77.6408356,
  "cityRadius": 5,
  "gameTimeActivities": false,
  "page": 0,
  "lastId": "",
  "sportId": ["SP5"],
  "booking": false,
  "date": ["2025-03-04T11:03:17.260Z"]
}

It returns a list of activities matching these filters. One such activity looks like:

{
  "userInfo": [
    {
      "profilePicUrl": "https://playov2.gumlet.io/profiles/redacted.511716.jpg",
      "fName": "Redacted",
      "lName": "",
      "karma": 2800
    },
    {
      "profilePicUrl": "https://playov2.gumlet.io/profiles/redacted-redacted.png",
      "fName": "redacted",
      "lName": "N",
      "karma": 499
    }
  ],
  "isPlayoGame": false,
  "skill": "Intermediate & above",
  "sportName": "Badminton",
  "shortListed": false,
  "joineeList": [
    "7f3cf298-3324-4fc2-96ad-b0f00093cd8f",
    "250572a2-555d-4a77-94f0-452142c08f81",
    "cc3b9eb6-a3b5-4c26-8605-0486fa000a4b",
    "8d5d4299-950b-4011-a7ac-b466b1c00e84",
    "235ae56d-6f4f-4106-9304-fb38e7d4add8"
  ],
  "isPlaypalPlaying": false,
  "lat": 12.976394040119704,
  "lng": 77.63644146986815,
  "location": "Game Theory - Double Road Indiranagar, Indiranagar",
  "joineeCount": 6,
  "status": -1,
  "sportsPlayingMode": {
    "name": "",
    "icon": ""
  },
  "maxPlayers": 7,
  "full": false,
  "price": 0,
  "startTime": "2025-03-04T13:30:00.000Z",
  "endTime": "2025-03-04T15:30:00.000Z",
  "minSkill": 3,
  "maxSkill": 5,
  "skillSet": true,
  "booking": false,
  "bookingId": "",
  "type": 0,
  "venueId": "82af038f-058c-4b2f-bc3d-3a47910d4f97",
  "venueName": "Game Theory - Double Road Indiranagar, Indiranagar",
  "activityType": "regular",
  "isOnline": false,
  "groupId": "",
  "groupName": "",
  "currencyTxt": "INR",
  "strictSkill": true,
  "date": "2025-03-04T00:00:00.000Z",
  "hostId": "redacted",
  "sportId": "SP5",
  "timing": 2,
  "id": "e2ee9f62-c9b6-472b-aea2-b0c52dd7c525",
  "distance": 0.5249236963063415,
  "courtInfo": "",
  "sponsored": false,
  "groups": []
}

Using the above response, I filtered for games where:

• full is false (This indicates that joineeCount == maxPlayer is not true, meaning spots are still available to join)
• startTime and endTime fall within 7-8PM IST

I also wanted to add a feature to send these details to Telegram for convenient notifications. I then vibe coded with Claude 3.7 to create a Python script to automate this whole process. Impressively, it produced a working script pretty much in a one-shot prompt, though I had to make a few minor tweaks. I quite like Simon Willison’s approach of using uv to build one-shot tools. Managing dependencies, virtual environments, etc. is still a pain point in Python, but using uv feels like magic by comparison.

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "click",
#     "requests",
#     "pytz",
#     "rich",
#     "python-dateutil",
#     "python-telegram-bot",
# ]
# ///

import click
import requests
import json
import datetime
import pytz
import os
import sys
from rich.console import Console
from rich.table import Table
from dateutil import parser
from telegram import Bot, InputMediaPhoto
from telegram.constants import ParseMode
from io import BytesIO
import asyncio

console = Console()

@click.command()
@click.option("--lat", default=12.9783692, help="Latitude for search")
@click.option("--lng", default=77.6408356, help="Longitude for search")
@click.option("--radius", default=50, help="City radius in km")
@click.option("--sport", default="SP5", help="Sport ID (default: SP5 for Badminton)")
@click.option("--start-time", default="19:00", help="Desired start time (24-hour format HH:MM)")
@click.option("--end-time", default="20:00", help="Desired end time (24-hour format HH:MM)")
@click.option("--timezone", default="Asia/Kolkata", help="Your timezone")
@click.option("--verbose", is_flag=True, help="Show detailed information including exact UTC/IST times")
@click.option("--include-full", is_flag=True, help="Include games that are full")
@click.option("--telegram", is_flag=True, help="Send results to Telegram")
@click.option("--telegram-token", envvar="TELEGRAM_BOT_TOKEN", help="Telegram Bot Token (or set TELEGRAM_BOT_TOKEN env var)")
@click.option("--telegram-chat-id", envvar="TELEGRAM_CHAT_ID", help="Telegram Chat ID (or set TELEGRAM_CHAT_ID env var)")
def find_games(lat, lng, radius, sport, start_time, end_time, timezone, verbose, include_full, telegram, telegram_token, telegram_chat_id):
    """Find available badminton games on Playo matching your criteria."""
    # Get today's date in the specified timezone
    local_tz = pytz.timezone(timezone)
    now = datetime.datetime.now(local_tz)
    today_date = now.strftime("%Y-%m-%dT%H:%M:%S.%fZ")

    # Parse desired time window
    try:
        desired_start = datetime.datetime.strptime(start_time, "%H:%M").time()
        desired_end = datetime.datetime.strptime(end_time, "%H:%M").time()
    except ValueError:
        console.print("[bold red]Error:[/bold red] Invalid time format. Please use HH:MM (24-hour format).")
        return

    console.print(f"[bold green]Searching for badminton games around your location...[/bold green]")
    console.print(f"Looking for games between [bold]{start_time}[/bold] and [bold]{end_time}[/bold] IST today")

    if verbose:
        console.print(f"[dim]Search parameters: lat={lat}, lng={lng}, radius={radius}km[/dim]")
        console.print(f"[dim]Current time in {timezone}: {now.strftime('%Y-%m-%d %H:%M:%S')}[/dim]")

    # Prepare API request
    url = "https://api.playo.io/activity-public/list/location"
    payload = {
        "lat": lat,
        "lng": lng,
        "cityRadius": radius,
        "gameTimeActivities": False,
        "page": 0,
        "lastId": "",
        "sportId": [sport],
        "booking": False,
        "date": [today_date]
    }

    headers = {
        "Content-Type": "application/json",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
    }

    try:
        response = requests.post(url, headers=headers, json=payload)
        response.raise_for_status()
        data = response.json()

        if data.get("requestStatus") != 1 or "data" not in data:
            console.print("[bold red]Error:[/bold red] Failed to get valid response from Playo API")
            return

        # Process activities
        activities = data["data"].get("activities", [])
        if not activities:
            console.print("[yellow]No badminton activities found for today[/yellow]")
            return

        # Filter activities based on criteria
        matching_games = []

        for activity in activities:
            # Convert UTC times to local timezone
            start_time_utc = parser.parse(activity["startTime"])
            end_time_utc = parser.parse(activity["endTime"])

            start_time_local = start_time_utc.astimezone(local_tz)
            end_time_local = end_time_utc.astimezone(local_tz)

            # Print all times in debug mode
            # console.print(f"DEBUG: {activity.get('location', 'Unknown')} - Start: {start_time_local.strftime('%H:%M')} IST (UTC: {start_time_utc.strftime('%H:%M')})")

            # Convert time objects correctly for comparison
            start_hour = start_time_local.hour
            start_minute = start_time_local.minute

            # Convert desired times to hours and minutes for easier comparison
            desired_start_hour = desired_start.hour
            desired_start_minute = desired_start.minute
            desired_end_hour = desired_end.hour
            desired_end_minute = desired_end.minute

            # Check if this game starts at 7PM (19:00) and ends at 8PM (20:00)
            is_time_match = False

            # Get duration in minutes
            duration_minutes = ((end_time_local.hour * 60 + end_time_local.minute) -
                               (start_time_local.hour * 60 + start_time_local.minute))

            # Check if start time is 7PM (with small tolerance)
            if (start_hour == desired_start_hour and
                start_minute >= desired_start_minute and
                start_minute < desired_start_minute + 10):  # Allow a small window of 10 minutes

                # Check if duration is approximately 1 hour (between 50-70 minutes)
                if 50 <= duration_minutes <= 70:
                    is_time_match = True

            # Check if there are available slots
            is_available = (
                not activity.get("full", True) and
                (activity.get("maxPlayers", 0) == -1 or
                 activity.get("joineeCount", 0) < activity.get("maxPlayers", 0))
            )

            # When verbose, print time details for each game to help debug
            if verbose:
                time_info = f"[dim]{activity.get('location', 'Unknown')} - Start: {start_time_local.strftime('%H:%M')} IST ({start_time_utc.strftime('%H:%M')} UTC), " + \
                           f"End: {end_time_local.strftime('%H:%M')} IST, Duration: {duration_minutes} min, " + \
                           f"Time match: {'Yes' if is_time_match else 'No'}, Available: {'Yes' if is_available else 'No'}[/dim]"
                console.print(time_info)

            # Both conditions must be true
            if is_time_match and is_available:
                matching_games.append({
                    "id": activity["id"],
                    "location": activity["location"],
                    "venue_name": activity.get("venueName", "N/A"),
                    "start": start_time_local.strftime("%I:%M %p"),
                    "end": end_time_local.strftime("%I:%M %p"),
                    "players": f"{activity.get('joineeCount', 0)}/{activity.get('maxPlayers', 'unlimited')}",
                    "host": activity.get("userInfo", [{}])[0].get("fName", "Unknown"),
                    "skill": activity.get("skill", "Any"),
                    "price": f"{activity.get('price', 0)} {activity.get('currencyTxt', 'INR')}"
                })

        # Display results
        if matching_games:
            table = Table(title=f"Available Badminton Games ({len(matching_games)} matches found)")

            table.add_column("Location", style="cyan")
            table.add_column("Time", style="green")
            table.add_column("Players", style="yellow")
            table.add_column("Host", style="magenta")
            table.add_column("Skill Level", style="blue")
            table.add_column("Link", style="bright_blue")

            for game in matching_games:
                table.add_row(
                    f"{game['venue_name']}",
                    f"{game['start']} - {game['end']}",
                    game["players"],
                    game["host"],
                    game["skill"],
                    f"https://playo.co/match/{game['id']}"
                )

            console.print(table)

            # Send to Telegram if requested
            if telegram:
                if not telegram_token or not telegram_chat_id:
                    console.print("[bold red]Error:[/bold red] Telegram token and chat ID are required for Telegram notifications")
                    console.print("[dim]Set them with --telegram-token and --telegram-chat-id or via environment variables[/dim]")
                else:
                    try:
                        send_to_telegram(matching_games, telegram_token, telegram_chat_id)
                        console.print("[green]Results sent to Telegram successfully![/green]")
                    except Exception as e:
                        console.print(f"[bold red]Error sending to Telegram:[/bold red] {e}")
        else:
            console.print("[yellow]No games found matching your criteria[/yellow]")
            if telegram and telegram_token and telegram_chat_id:
                try:
                    asyncio.run(send_telegram_message(
                        "No badminton games found matching your criteria for today.",
                        telegram_token,
                        telegram_chat_id
                    ))
                    console.print("[green]Empty results notification sent to Telegram[/green]")
                except Exception as e:
                    console.print(f"[bold red]Error sending to Telegram:[/bold red] {e}")

    except requests.RequestException as e:
        console.print(f"[bold red]Error:[/bold red] Failed to connect to Playo API: {e}")
    except json.JSONDecodeError:
        console.print("[bold red]Error:[/bold red] Failed to parse API response")
    except Exception as e:
        console.print(f"[bold red]Error:[/bold red] An unexpected error occurred: {e}")

def send_to_telegram(games, token, chat_id):
    """Send game information to Telegram as a nicely formatted message."""
    if not games:
        return

    # Create a formatted message for Telegram
    message = "🏸 *Available Badminton Games* 🏸\n\n"

    for i, game in enumerate(games, 1):
        message += f"*{i}. {game['venue_name']}*\n"
        message += f"⏰ {game['start']} - {game['end']}\n"
        message += f"👥 Players: {game['players']}\n"
        message += f"👤 Host: {game['host']}\n"
        message += f"🎯 Skill: {game['skill']}\n"
        message += f"🔗 [Join Game](https://playo.co/match/{game['id']})\n\n"

    # Send the message
    asyncio.run(send_telegram_message(message, token, chat_id))

async def send_telegram_message(message, token, chat_id):
    """Send a message to Telegram using the Bot API."""
    bot = Bot(token=token)
    await bot.send_message(
        chat_id=chat_id,
        text=message,
        parse_mode=ParseMode.MARKDOWN,
        disable_web_page_preview=False
    )


if __name__ == "__main__":
    find_games()

The script outputs a beautiful output:

Telegram:

Scheduling#

I wanted this script to run reliably every day and used GitHub Actions for that. GitHub Actions felt like the path of least resistance as I didn’t have to worry about keeping a server running or getting alerts if something crashed. For a small personal script like this, it was the perfect “set it and forget it” solution.

name: Badminton Game Checker Base
on:
  schedule:
    # Run Monday to Friday at 12:00 PM IST (6:30 AM UTC)
    - cron: "30 6 * * 1-5"

jobs:
  check-games:
    runs-on: ubuntu-latest

    env:
      TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
      TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }}
      LATITUDE: ${{ inputs.latitude }}
      LONGITUDE: ${{ inputs.longitude }}
      RADIUS: ${{ inputs.radius }}
      SPORT_ID: ${{ inputs.sport_id }}
      TIMEZONE: ${{ inputs.timezone }}
      START_TIME: ${{ inputs.start_time }}
      END_TIME: ${{ inputs.end_time }}

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install uv

      - name: Run game check
        run: |
          echo "Checking for games from $START_TIME to $END_TIME"
          uv run finder.py \
            --lat "$LATITUDE" \
            --lng "$LONGITUDE" \
            --radius "$RADIUS" \
            --sport "$SPORT_ID" \
            --timezone "$TIMEZONE" \
            --start-time "$START_TIME" \
            --end-time "$END_TIME" \
            --telegram
        env:
          TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
          TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }}

I used GitHub Actions inputs to configure the variables for my script. Found this feature to be quite neat for scheduling different crons for weekday/weekends.

Summary#

For small quality-of-life improvements - solving your own specific problems with custom scripts tailored exactly to your needs - gotta love the LLMs man. We’re gonna see more and more of such “personal tooling” in future as the entry to barrier for coding is lowered with LLMs. The democratization of coding through LLMs means people (even non-technical ones) can focus on “describing” the problem well, rather than struggling with implementation details. Being able to articulate what you want clearly becomes the primary skill - yes, it’s a skill issue if you can’t prompt well, but it’s far more accessible than learning programming from scratch.

Fin!

]]> Karan Sharma https://www.divyamohan.com/open-source-security/ https://www.divyamohan.com/open-source-security/ Fri, 28 Feb 2025 04:51:07 GMT A panel discussion between Aeva Black, Mike Bursell, and Nelson Batsford at State of Open Con 2025 in London moderated by Divya Mohan

The goal of this discussion is two-fold:

Firstly, supply chain security remains a mystery despite the many tools, frameworks, and ecosystems we've built around it - open source and otherwise. Even if we scope our discussion to include only the supply chain security of the open source components we employ in our workloads, how many of us can claim that we truly are aware of all our dependencies & can mitigate all the upstream risks associated with them? Especially in these times when malicious actors are becoming more sophisticated & AI-generated code is becoming the norm? Are these even a part of organisational discussions or considerations while adopting open source tools or technologies today?

Secondly, how do regulations, for example, the European Union's CRA, UK’s Cyber Security and Resilience Bill, etc., impact conversations around supply chain security? Are they helpful in improving the support we extend to the creators and maintainers of the software we depend on?

With this panel, we aimed to explore some of the upstream and downstream risks associated with open source software security today while also seeking to set the ball rolling on a collaborative dialogue toward addressing some of the current challenges and providing insights into future implications

]]> Divya Mohan Open Source tech policy https://www.evalapply.org/posts/writing-practices-to-10x-engineering/index.html https://www.evalapply.org/posts/writing-practices-to-10x-engineering/index.html Fri, 28 Feb 2025 00:00:00 GMT Aditya Athalye meta whyto howto writing systems culture tools_for_thought programming https://www.evalapply.org/posts/systems-approach-to-infrastructure-as-code/index.html https://www.evalapply.org/posts/systems-approach-to-infrastructure-as-code/index.html Tue, 18 Feb 2025 00:00:00 GMT Aditya Athalye howto systems infrastructure devops architecture https://nadh.in/blog/deepseek-ai-sovereignty-india/ https://nadh.in/blog/deepseek-ai-sovereignty-india/ Wed, 29 Jan 2025 00:00:00 GMT Along came DeepSeek-R1^[1] last week, an open-source large language model (LLM) reportedly rivaling OpenAI’s top offerings, sending shockwaves through the industry and generating much excitement in the tech world. It apparently started as a side project at a Chinese hedge fund before being spun out. Its efficacy, combined with claims of being built at a fraction of the cost and hardware requirements, has seriously challenged BigAI’s notion that “foundation models” demand astronomical investments. I have personally been playing around with R1 and have found it to be excellent at writing code. Speaking of foundation models, one rarely hears that term anymore; unsurprising, given that foundation is now commodity. Building a foundation-level LLM was once touted as the cornerstone of AI sovereignty, but that rhetoric has also waned. Much has changed regarding the idea of AI sovereignty.

]]> Kailash Nadh https://ibcomputing.com/deepseek-r1/ https://ibcomputing.com/deepseek-r1/ Wed, 22 Jan 2025 17:28:59 GMT Artificial intelligence continues to evolve at a rapid pace, and DeepSeek R1 is at the forefront of this transformation. Building on the success of DeepSeek V3, the R1 model introduces groundbreaking features like DeepThinking, an advanced reasoning engine designed to enhance problem-solving capabilities. Unlike proprietary systems, DeepSeek R1 is fully open-source, released under the MIT License, allowing for unrestricted commercial use, modification, and distribution. Combined with its robust API offerings, DeepSeek R1 is poised to redefine how businesses and developers leverage AI for complex tasks.

In this article, we’ll dive into the key features of DeepSeek R1, explore the innovative DeepThinking technology, and discuss how its API integration makes it a game-changer for developers and enterprises.

---

Use DeepSeek

Go to chat.deepseek.com  for trying deepseek for free. there is no limit for the usage now. also their api is less costly.

What is DeepSeek R1?

DeepSeek R1 is the latest iteration of DeepSeek’s AI language models, designed to tackle even more complex and nuanced tasks. With a focus on reasoning, contextual understanding, and adaptability, R1 builds on the strengths of its predecessors while introducing new capabilities that set it apart from competitors like OpenAI’s GPT-4.

Key features of DeepSeek R1 include:

• DeepThinking: An advanced reasoning engine that enhances problem-solving and decision-making.
• Open-Source Licensing: Released under the MIT License, allowing for free commercial use, modification, and distribution16.
• API Integration: Seamless integration with existing systems for developers and enterprises.
• Scalability: Designed to handle large-scale applications with ease.
• Ethical AI: Continued focus on reducing biases and promoting inclusivity.

 

---

DeepThinking: The Brain Behind DeepSeek R1

One of the most exciting innovations in DeepSeek R1 is DeepThinking, a reasoning engine that enhances the model’s ability to reason, analyze, and solve problems. Unlike traditional AI models that rely on pattern recognition, DeepThinking enables R1 to:

1. Break Down Complex Problems: DeepThinking allows the model to dissect intricate problems into smaller, manageable components, making it ideal for tasks like coding, research, and strategic planning14.
2. Simulate Human-Like Reasoning: By incorporating advanced algorithms, DeepThinking enables R1 to simulate human-like reasoning, resulting in more accurate and contextually relevant outputs11.
3. Adapt to New Scenarios: DeepThinking ensures that R1 can adapt to unfamiliar situations, making it a versatile tool for industries like healthcare, finance, and education4.

For example, in a coding scenario, DeepThinking allows R1 to not only generate code but also debug and optimize it, providing a comprehensive solution that goes beyond basic suggestions

Open-Source Licensing and Community Impact

DeepSeek R1 is released under the MIT License, which grants users the freedom to use, modify, and distribute the model for both personal and commercial purposes without restrictions16. This open-source approach fosters collaboration and innovation within the AI community, enabling researchers and developers to build upon DeepSeek R1’s capabilities.

Additionally, DeepSeek has open-sourced six distilled models (ranging from 1.5B to 70B parameters) based on Qwen and Llama architectures. These smaller models retain the reasoning capabilities of R1, making them suitable for resource-constrained environments14.

API Usage: Empowering Developers and Enterprises

DeepSeek R1’s API integration is another standout feature, offering developers and businesses a powerful tool to incorporate AI into their workflows. Here’s how the API can be used:

1. Seamless Integration

The DeepSeek R1 API is designed for easy integration with existing systems, allowing developers to quickly deploy AI capabilities without extensive reconfiguration26.

2. Cost-Effective Pricing

DeepSeek R1’s API is significantly more affordable than competitors like OpenAI, with pricing at $0.55permillioninputtokens\ast \ast and\ast \ast$2.19 per million output tokens28.

3. Real-World Applications

• Healthcare: Use the API to analyze patient data, generate reports, and assist in diagnosis.
• Finance: Integrate R1 for risk assessment, market analysis, and automated trading.
• Education: Create personalized learning experiences and automate administrative tasks.
• Customer Support: Deploy AI-powered chatbots for efficient and accurate customer service

Conclusion

DeepSeek R1 represents a significant leap forward in AI technology, combining the power of DeepThinking with seamless API integration and open-source accessibility. Whether you’re a developer looking to build cutting-edge applications or an enterprise seeking to optimize workflows, DeepSeek R1 offers the tools and capabilities to meet your needs.

With its focus on reasoning, adaptability, and ethical AI, DeepSeek R1 is not just a competitor to GPT-4—it’s a step toward the future of artificial intelligence.

 

The post DeepSeek R1: Revolutionizing AI with Open Source and DeepThinking appeared first on IB Computing.

]]> Mujeeb cpy AI News artificial intelligence https://ravidwivedi.in/posts/luxembourg-visa-process/ https://ravidwivedi.in/posts/luxembourg-visa-process/ Tue, 21 Jan 2025 10:45:57 GMT In 2024, I was sponsored by The Document Foundation (TDF) to attend the LibreOffice annual conference in Luxembourg from the 10th to the 12th of October. Being an Indian passport holder, I needed a visa to visit Luxembourg. However, due to my Kenya trip coming up in September, I ran into a dilemma: whether to apply before or after the Kenya trip.

To obtain a visa, I needed to submit my application with VFS Global (and not with the Luxembourg embassy directly). Therefore, I checked the VFS website for information on processing time, which says:

As a rule, the processing time of an admissible Schengen visa application should not exceed 15 calendar days (from the date the application is received at the Embassy).

It also mentions:

If the application is received less than 15 calendar days before the intended travel date, the Embassy can deem your application inadmissible. If so, your visa application will not be processed by the Embassy and the application will be sent back to VFS along with the passport.

If I applied for the Luxembourg visa before my trip, I would run the risk of not getting my passport back in time, and therefore missing my Kenya flight. On the other hand, if I waited until after returning from Kenya, I would run afoul of the aforementioned 15 working days needed by the embassy to process my application.

I had previously applied for a Schengen visa for Austria, which was completed in 7 working days. My friends who had been to France told me they got their visa decision within a week. So, I compared Luxembourg’s application numbers with those of other Schengen countries. In 2023, Luxembourg received 3,090 applications from India, while Austria received 39,558, Italy received 52,332 and France received 176,237. Since Luxembourg receives a far fewer number of applications, I expected the process to be quick.

Therefore, I submitted my visa application with VFS Global in Delhi on the 5th of August, giving the embassy a month with 18 working days before my Kenya trip. However, I didn’t mention my Kenya trip in the Luxembourg visa application.

For reference, here is a list of documents I submitted:

• Passport
• Photocopy of passport data pages
• Visa application form
• One photograph
• Visa appointment confirmation
• Cover letter
• Return flight reservations
• Hotel bookings
• Invitation letter from the conference organizer, TDF
• Confirmation from The Luxembourg Convention Bureau G.I.E - the venue
• Last three months bank account statement with bank seal
• Travel insurance
• Income Tax Return Statement
• Monthly payslips for the last three months

I submitted ‘flight reservations’ instead of ‘flight tickets’. It is because, in case of visa rejection, I would have lost a significant amount of money if I booked confirmed flight tickets. The embassy also recommends the same. After the submission of documents, my fingerprints were taken.

The expenses for the visa application were as follows:

Service Description	Amount (INR)
Visa Fee	8,114
VFS Global Fee	1,763
Courier	800
Total	10,677

Going by the email notifications I received from VFS, my application reached the Luxembourg embassy the next day. Fast-forward to the 27th of August — 14th day of my visa application. I had already booked my flight ticket to Nairobi for the 4th of September, but my passport was still with the Luxembourg embassy, and I hadn’t heard back. In addition, I also obtained Kenya’s eTA and got vaccinated for Yellow Fever, a requirement to travel to Kenya.

In order to check on my application status, I gave the embassy a phone call, but missed their calling window, which was easy to miss since it was only 1 hour - 12:00 to 1:00 PM. So, I dropped them an email explaining my situation. At this point, I was already wondering whether to cancel the Kenya trip or the Luxembourg one, if I had to choose.

After not getting a response to my email, I called them again the next day. The embassy told me they would look into it and asked me to send my flight tickets over email. One week to go before my flight now.

I followed up with the embassy on the 30th by a phone call, and the person who picked up the call told me that my request had already been forwarded to the concerned department and is under process. They asked me to follow up on Monday, 2nd September.

During the visa process, I was in touch with three other Indian attendees.¹ In the meantime, I got to know that all of them had applied for a Luxembourg visa by the end of the month of August.

Back to our story, over the next two days, the embassy closed for the weekend. I began weighing my options. On one hand, I could cancel the Kenya trip and hope that Luxembourg goes through. Even then, Luxembourg wasn’t guaranteed as the visa could get rejected, so I might have ended up missing both the trips. On the other hand, I could cancel the Luxembourg visa application and at least be sure of going to Kenya. However, I thought it would make Luxembourg very unlikely because it didn’t leave 15 working days for the embassy to process my visa after returning from Kenya. I also badly wanted to attend the LibreOffice conference because I couldn’t make it two years ago. Therefore, I chose not to cancel my Luxembourg visa application. I checked with my travel agent and learned that I could cancel my Nairobi flight before September 4th for a cancelation fee of approximately 7,000 INR.

On the 2nd of September, I was a bit frustrated because I hadn’t heard anything from the embassy regarding my request. Therefore, I called the embassy again. They assured me that they would arrange a call for me from the concerned department that day, which I did receive later that evening. During the call, they offered to return my passport via VFS the next day and asked me to resubmit it after returning from Kenya. I immediately accepted the offer and was overjoyed, as it would enable me to take my flight to Nairobi without canceling my Luxembourg visa application. However, I didn’t have the offer in writing, so it wasn’t clear to me how I would collect my passport from VFS. The next day, I would receive it when I would be on my way to VFS in the form of an email from the embassy which read:

Dear Mr. Dwivedi,

We acknowledge the receipt of your email.

As you requested, we are returning your passport exceptionally through VFS, you can collect it directly from VFS Delhi Center between 14:00-17:00 hrs, 03 Sep 2024. Kindly bring the printout of this email along with your VFS deposit receipt and Original ID proof.

Once you are back from your trip, you can redeposit the passport with VFS Luxembourg for our processing.

With best regards,
Consular Section

GRAND DUCHY OF LUXEMBOURG
Embassy in New Delhi

I took a printout of the email and submitted it to VFS to get my passport. This seemed like a miracle - just when I lost all hope of making it to my Kenya flight and was mentally preparing myself to miss it, I got my passport back “exceptionally” and now I had to mentally prepare again for Kenya. I had never heard of an embassy returning passport before completing the visa process before. The next day, I took my flight to Nairobi as planned. In case you are interested, I have written two blog posts on my Kenya trip - one on the OpenStreetMap conference in Nairobi and the other on my travel experience in Kenya.

After returning from Kenya, I resubmitted my passport on the 17th of September. Fast-forward to the 25th of September; I didn’t hear anything from the embassy about my application process. So, I checked with TDF to see whether the embassy reached out to them. They told me they confirmed my participation and my hotel booking to the visa authorities on the 19th of September (6 days ago). I was wondering what was taking so long after the verification.

On the 1st of October, I received a phone call from the Luxembourg embassy, which turned out to be a surprise interview. They asked me about my work, my income, how I came to know about the conference, whether I had been to Europe before, etc. The call lasted around 10 minutes. At this point, my travel date - 8th of October - was just two working days away as the 2nd of October was off due to Gandhi Jayanti and 5th and 6th October were weekends, leaving only the 3rd and the 4th. I am not sure why the embassy saved this for the last moment, even though I submitted my application 2 months ago. I also got to know that one of the other Indian attendees missed the call due to being in their college lab, where he was not allowed to take phone calls. Therefore, I recommend that the embassy agree on a time slot for the interview call beforehand.

Visa decisions for all the above-mentioned Indian attendees were sent by the embassy on the 4th of October, and I received mine on the 5th. For my travel date of 8th October, this was literally the last moment the embassy could send my visa. The parcel contained my passport and a letter. The visa was attached to a page in the passport. I was happy that my visa had been approved. However, the timing made my task challenging. The enclosed letter stated:

Subject: Your Visa Application for Luxembourg

Dear Applicant,

We would like to inform you that a Schengen visa has been granted for the 8-day duration from 08/10/2024 to 30/10/2024 for conference purposes in Luxembourg.

You are requested to report back to the Embassy of Luxembourg in New Delhi through an email (email address redacted) after your return with the following documents:

• Immigration Stamps (Entry and Exit of Schengen Area)
• Restaurant Bills
• Shopping/Hotel/Accommodation bills

Failure to report to the Embassy after your return will be taken into consideration for any further visa applications.

I understand the embassy wanting to ensure my entry and exit from the Schengen area during the visa validity period, but found the demand for sending shopping bills excessive. Further, not everyone was as lucky as I was as it took a couple of days for one of the Indian attendees to receive their visa, delaying their plan. Another attendee had to send their father to the VFS center to collect their visa in time, rather than wait for the courier to arrive at their home.

Foreign travel is complicated, especially for the citizens of countries whose passports and currencies are weak. Embassies issuing visas a day before the travel date doesn’t help. For starters, a last-minute visa does not give enough time for obtaining a forex card as banks ask for the visa. Further, getting foreign currency (Euros in our case) in cash with a good exchange rate becomes difficult. As an example, for the Kenya trip, I had to get US Dollars at the airport due to the plan being finalized at the last moment, worsening the exchange rate. Back to the current case, the flight prices went up significantly compared to September, almost doubling. The choice of airlines also got narrowed, as most of the flights got booked by the time I received my visa. With all that said, I think it was still better than an arbitrary rejection.

Credits: Contrapunctus, Badri, Fletcher, Benson, and Anirudh for helping with the draft of this post.

---

1. Thanks to Sophie, our point of contact for the conference, for putting me in touch with them. ↩︎

]]> Ravi Dwivedi https://www.prashanthudupa.com/the-feeling-of-thoughts/ https://www.prashanthudupa.com/the-feeling-of-thoughts/ Mon, 20 Jan 2025 16:25:36 GMT Prashanth Udupa Insight Philosophy https://www.learningfromdata.zingg.ai/p/the-open-source-identity-resolution https://www.learningfromdata.zingg.ai/p/the-open-source-identity-resolution Tue, 07 Jan 2025 15:32:54 GMT More than a decade ago, in the year 2013, the identity resolution problem hit me. As a niche consulting company focused on data and AI, we were tasked with integrating customer records from disparate systems. Our project involved setting up a Hadoop cluster on Amazon EC2 and building out the analytics and reporting for our client. This was the time when services like Elastic Map Reduce were about to get launched. It felt like an engineering feat that we could run Hadoop on the cloud through custom scripts. In a way, we were actually mimicking today’s serverless approaches.

We built data extraction, transformation, and loading pipelines along with the reporting using Sqoop, Pig, Hive, and Cascading. But, as we got along the project, we faced a critical issue. Even when we got all the data in one place, it was extremely challenging to identify the real world entity represented by the varying records from different systems. Since the records did not have any common identifiers and they also had all sorts of variations, unifying the data became the biggest issue for us on that project.

Once I became aware of the problem, I started seeing it in other projects. Even when I was doing different work, the problem kind of never left me. How do we say that the records are a match? How do we scale this out? How do we run it for any entity which can appear for an enterprise? For the programmer in me, it became a thrill ride of problem solving with lots of first principle thinking. The more I worked on it, the more challenges I became aware of. Eventually, I got completely immersed and stopped taking other consulting work to focus on building a generic solution.

When I showed an early version to my friend and mentor Joydeep Sen Sarma, creator of Apache Hive, he encouraged me to open source it. As an active consumer of amazing open source technologies, it was a great way to contribute back. So I latched on to the idea. Here is what I wrote when I published Zingg:

I also feel that open source Zingg is more powerful than closed source, because then it is up to the community to put it to use in more ways than I have seen and could ever anticipate. It is also a promise to work closely with different people and collaborate, something I am very sincerely hoping will happen!

I was convinced there were more people like me, but it was not clear if they would find me. Fast forward to today - the Zingg open source community is 700+ members strong. Turns out what felt like an esoteric problem to me is a strongly felt pain for enterprises. Starting with a single person, Zingg now has a founding team that cares deeply about entity resolution. Zingg resolves billions of records every month and our users and customers come from diverse industries like healthcare, insurance, government sector, non profits as well as startups. This love and support is driving us to innovate more, and we are building amazing stuff to run production loads with a persistent ZINGG_ID along with incremental flows. We continue to work extensively on accuracy and performance. Zingg also supports diverse deployments on Microsoft Fabric, Snowflake, AWS Glue, Databricks, and others.

All this would not have been possible without our community members, well wishers and supporters. Thank you all, you have made this journey well worth it

!

]]> Sonal Goyal https://mrkaran.dev/posts/cleanup-obsidian/ https://mrkaran.dev/posts/cleanup-obsidian/ Fri, 03 Jan 2025 04:45:00 GMT category: tags: - tag1 - tag2 status: priority: description: """ def clean_llm_response(self, response_text): """Clean up the LLM response to ensure proper YAML.""" # Remove yaml code block markers if present response_text = response_text.strip() if response_text.startswith('```yaml'): response_text = response_text.split('\n', 1)[1] if response_text.endswith('```'): response_text = response_text.rsplit('\n', 1)[0] return response_text.strip() def process_note(self, file_path): """Process a single note file.""" try: with open(file_path, 'r', encoding='utf-8') as f: content = f.read() # Extract existing frontmatter and content existing_frontmatter, main_content = self.extract_existing_frontmatter(content) # Generate and execute prompt response = self.model.prompt(self.generate_prompt(main_content)) response_text = self.clean_llm_response(response.text()) try: new_frontmatter = yaml.safe_load(response_text) if not isinstance(new_frontmatter, dict): print(f"Warning: Invalid response format for {file_path.name}") new_frontmatter = {} except yaml.YAMLError as e: print(f"YAML parsing error for {file_path.name}") print(f"Response text was:\n{response_text}") raise e # Merge with existing frontmatter, preferring existing values merged_frontmatter = {**new_frontmatter, **existing_frontmatter} # Add date if not present if 'date' not in merged_frontmatter: merged_frontmatter['date'] = datetime.date.today().isoformat() # Generate new note content new_content = "---\n" new_content += yaml.dump(merged_frontmatter, sort_keys=False, allow_unicode=True) new_content += "---\n\n" new_content += main_content.strip() # Write back to file with open(file_path, 'w', encoding='utf-8') as f: f.write(new_content) print(f"✓ Processed: {file_path.name}") except Exception as e: print(f"✗ Error processing {file_path.name}: {str(e)}") def process_vault(self): """Process all markdown files in the vault.""" print("Starting Obsidian vault cleanup...") for file_path in self.notes_dir.glob('**/*.md'): self.process_note(file_path) print("\nVault cleanup completed!") def main(): # Set up the model key if not already configured model = llm.get_model("claude-3.5-sonnet") if not hasattr(model, 'key'): api_key = os.getenv('ANTHROPIC_API_KEY') if not api_key: raise ValueError("Please set ANTHROPIC_API_KEY environment variable") model.key = api_key # Initialize and run the processor notes_dir = "/Users/karan/Notes/Obsidian/The Wall/Notes" processor = ObsidianNoteProcessor(notes_dir) processor.process_vault() if __name__ == "__main__": main() The script is pretty straightforward – it reads each markdown file, extracts any existing frontmatter (because I don’t want to lose that!), and then asks Claude to analyze the content and generate appropriate frontmatter. It adds stuff like title, category, tags, status, priority. What I love about this approach is that it’s contextual. Unlike regex-based approaches or keyword matching, the LLM actually understands what the note is about and can categorize it properly. A note about “Setting up BTRFS on Arch” automatically gets tagged with “linux”, “filesystem”, “arch” without me having to maintain a predefined list of tags. The categorization is probably better than what I’d have done manually at 2 AM while organizing my notes! Fin!]]> My Obsidian vault has gotten quite messy over time. I’ve been dumping notes without proper frontmatter, tags were all over the place, and some notes didn’t even have proper titles! I needed a way to clean this up without spending hours manually organizing everything.

I’d been playing around with Claude’s API lately, and thought – hey, why not use an LLM to analyze my notes and add proper frontmatter? After all, that’s what these AI models are good at – understanding context and categorizing stuff.

I wrote a small Python script using the llm library (which is pretty neat btw) to do just this. Here’s what it looks like:

import llm
import os
import yaml
import datetime
from pathlib import Path
import re

class ObsidianNoteProcessor:
    def __init__(self, notes_dir, model_name="claude-3.5-sonnet"):
        self.notes_dir = Path(notes_dir)
        self.model = llm.get_model(model_name)
        
    def extract_existing_frontmatter(self, content):
        """Extract existing frontmatter if present."""
        frontmatter_pattern = r'^---\n(.*?)\n---\n'
        match = re.match(frontmatter_pattern, content, re.DOTALL)
        
        if match:
            try:
                return yaml.safe_load(match.group(1)), content[match.end():]
            except yaml.YAMLError:
                return {}, content
        return {}, content

    def generate_prompt(self, content):
        """Generate a prompt for the LLM to analyze the note content."""
        return f"""Analyze the following note content and extract/infer the following properties:
1. A clear title (if not present, generate from content)
2. Relevant categories based on the content
3. Appropriate tags (include 'inbox' if content seems draft-like)
4. Status (Draft/In Progress/Complete) based on content completeness
5. Priority (Low/Medium/High) based on content importance
6. A brief description summarizing the content

Note content:
{content}

Return ONLY the YAML frontmatter without any code block markers. Use this exact format (omit fields if not applicable):
title: <title>
category: <category>
tags:
  - tag1
  - tag2
status: <status>
priority: <priority>
description: <description>"""

    def clean_llm_response(self, response_text):
        """Clean up the LLM response to ensure proper YAML."""
        # Remove yaml code block markers if present
        response_text = response_text.strip()
        if response_text.startswith('```yaml'):
            response_text = response_text.split('\n', 1)[1]
        if response_text.endswith('```'):
            response_text = response_text.rsplit('\n', 1)[0]
        return response_text.strip()

    def process_note(self, file_path):
        """Process a single note file."""
        try:
            with open(file_path, 'r', encoding='utf-8') as f:
                content = f.read()
            
            # Extract existing frontmatter and content
            existing_frontmatter, main_content = self.extract_existing_frontmatter(content)
            
            # Generate and execute prompt
            response = self.model.prompt(self.generate_prompt(main_content))
            response_text = self.clean_llm_response(response.text())
            
            try:
                new_frontmatter = yaml.safe_load(response_text)
                if not isinstance(new_frontmatter, dict):
                    print(f"Warning: Invalid response format for {file_path.name}")
                    new_frontmatter = {}
            except yaml.YAMLError as e:
                print(f"YAML parsing error for {file_path.name}")
                print(f"Response text was:\n{response_text}")
                raise e
            
            # Merge with existing frontmatter, preferring existing values
            merged_frontmatter = {**new_frontmatter, **existing_frontmatter}
            
            # Add date if not present
            if 'date' not in merged_frontmatter:
                merged_frontmatter['date'] = datetime.date.today().isoformat()
            
            # Generate new note content
            new_content = "---\n"
            new_content += yaml.dump(merged_frontmatter, sort_keys=False, allow_unicode=True)
            new_content += "---\n\n"
            new_content += main_content.strip()
            
            # Write back to file
            with open(file_path, 'w', encoding='utf-8') as f:
                f.write(new_content)
                
            print(f"✓ Processed: {file_path.name}")
            
        except Exception as e:
            print(f"✗ Error processing {file_path.name}: {str(e)}")

    def process_vault(self):
        """Process all markdown files in the vault."""
        print("Starting Obsidian vault cleanup...")
        
        for file_path in self.notes_dir.glob('**/*.md'):
            self.process_note(file_path)
        
        print("\nVault cleanup completed!")

def main():
    # Set up the model key if not already configured
    model = llm.get_model("claude-3.5-sonnet")
    if not hasattr(model, 'key'):
        api_key = os.getenv('ANTHROPIC_API_KEY')
        if not api_key:
            raise ValueError("Please set ANTHROPIC_API_KEY environment variable")
        model.key = api_key
    
    # Initialize and run the processor
    notes_dir = "/Users/karan/Notes/Obsidian/The Wall/Notes"
    processor = ObsidianNoteProcessor(notes_dir)
    processor.process_vault()

if __name__ == "__main__":
    main()

The script is pretty straightforward – it reads each markdown file, extracts any existing frontmatter (because I don’t want to lose that!), and then asks Claude to analyze the content and generate appropriate frontmatter. It adds stuff like title, category, tags, status, priority.

What I love about this approach is that it’s contextual. Unlike regex-based approaches or keyword matching, the LLM actually understands what the note is about and can categorize it properly. A note about “Setting up BTRFS on Arch” automatically gets tagged with “linux”, “filesystem”, “arch” without me having to maintain a predefined list of tags. The categorization is probably better than what I’d have done manually at 2 AM while organizing my notes!

Fin!

]]> Karan Sharma https://www.evalapply.org/posts/hello-quarter-two-of-the-21st-century/index.html https://www.evalapply.org/posts/hello-quarter-two-of-the-21st-century/index.html Wed, 01 Jan 2025 00:00:00 GMT Aditya Athalye meta riff https://programmerlife1.wordpress.com/2024/12/23/tholkappiyam-discussion/ https://programmerlife1.wordpress.com/2024/12/23/tholkappiyam-discussion/ Mon, 23 Dec 2024 17:19:27 GMT check point => Jan 20th.]]> Discussion – Dec 23, 2024
Date and Time

Dec 7, 2024 20:30-22:00 IST
Members Joined

Boopalan S
Syed Jafer
Hariharan

Topics discussed

Cosmetics:

Where to maintain the code base ?

Contribution Related:

Language to be maintained in the code ? TBD

Codebase:

1. First Documentation – Just The Docs

-> check point => Jan 20th.

2. Code Structure – TBD
3. Parallel Test Cases – Pytest -> Github Actions [Lint, Test] [Precommit hooks]
4. To collect more tc from Satyaraj Sir

]]> Hariharan Uncategorized https://mrkaran.dev/posts/2024/ https://mrkaran.dev/posts/2024/ Mon, 23 Dec 2024 06:30:00 GMT Mattermost bridge toru - Go modules proxy with caching junbi - Server Setup and Hardening Tool Ovenly Delights - Small bakery shop website nomcfg - Nomad config generator clx - Generate CLI commands using AI for common ops Started working on a log analytics app - full focus on that in 2025. Read more This year has been truly transformative, bringing together personal joy, professional growth, and exciting adventures. Looking forward to what 2025 has in store!]]> 2024 was indeed an important year for me as it marked several significant milestones. Quite happy with how this year was! Here’s my reflection on this memorable year.

Life#

• Got married to the prettiest and dearest Saumya 💗
• Did my first international trip, exploring Europe
• Bought a fun toy - Maruti Jimny 4x4
• Relocated to Bangalore after working from home for 4+ years since Covid
• Attended several amazing concerts:
  • Indian Ocean
  • Parvaaz
  • Blackstratblues
  • Anand Bhaskar Collective
  • Bandland

Travel#

• Switzerland
• Italy
• Austria
• Binsar
• Cochin
• Ajmer
• Ranthambore
• Pondicherry
• Chennai

Projects#

• Released v1.0.0 of Doggo - It hit frontpage of HN as well!
• Built an expense tracker app - Gullak
• Made a lot of small utility apps:
  • lil - URL shortener
  • silencer - Prometheus alerts <> Mattermost bridge
  • toru - Go modules proxy with caching
  • junbi - Server Setup and Hardening Tool
  • Ovenly Delights - Small bakery shop website
  • nomcfg - Nomad config generator
  • clx - Generate CLI commands using AI for common ops
• Started working on a log analytics app - full focus on that in 2025. Read more

This year has been truly transformative, bringing together personal joy, professional growth, and exciting adventures. Looking forward to what 2025 has in store!

]]> Karan Sharma https://shrirangkahale.com/donate/ https://shrirangkahale.com/donate/ Fri, 13 Dec 2024 00:00:00 GMT Shrirang Kahale https://programmerlife1.wordpress.com/2024/12/03/kanchilug-engine-mode/ https://programmerlife1.wordpress.com/2024/12/03/kanchilug-engine-mode/ Tue, 03 Dec 2024 18:29:55 GMT டிச, 3 2024

காஞ்சிபுரம் லினக்ஸ் பயனர் குழுவின் ஒருங்கிணைப்பாளர் பொறுப்பிலிருந்து விடுபடுவதாக திரு. பரமேஸ்வர் அருணாச்சலம் அவர்கள் வாராந்திர கூட்டத்திலும் ,மடல் பட்டியலிலும் அறிவித்திருந்தார்.

நான் காஞ்சி லினக்ஸ் பயனர் குழுவில் இணைந்ததில் இருந்து பரமேஸ்வர் அவர்கள் சிறப்பாக வாராந்திர மற்றும் மாதாந்திர கூட்டத்தினை நடத்திவந்தார்.

வரும் வாரங்களில் அவரைப் போலவே சிறப்பாக வாரந்திர மற்றும் மாதாந்திர கூட்டங்களை நடத்தும் ஒருங்கிணைப்பாளராக நான் பொறுபேற்றுக்கொள்ள இருக்கிறேன்.

சில பொறுப்புகளை ஏற்று நடத்துதலில் நல்ல நிர்வாக திறனை பெறலாம். அப்படி நான் கற்பனவற்றையும் எழுதுகிறேன் வலைப்பதிவுகளாக…

]]> Hariharan Uncategorized https://programmerlife1.wordpress.com/2024/12/01/curlftpfs-a-glance/ https://programmerlife1.wordpress.com/2024/12/01/curlftpfs-a-glance/ Sun, 01 Dec 2024 11:33:37 GMT CurlFtpFS – A FTP filesystem based on cURL and FUSE

ஆக என்ன பயன்?

ஒரு ftp தளத்திலிருந்து கோப்பு மேலாளர்(nautilus,nemo…etc) மூலமாகவே ஒரு தளத்தினை வன்வட்டில் உள்ளது போலவே அணுக இயலும்.

எவ்வாறு நிறுவுவது?

sudo apt install curlftpfs

எனும் கட்டளையை இயக்கி நிறுவலாம்.

எவ்வாறு பயன்படுத்துவது?

முதலில் நமது கோப்பு அமைப்பில் நாம் பயன்படுத்தபோகிற ftp தளத்திற்கு ஒரு கோப்புறை உருவாக்குக (கோப்புறைக்கு 777 அனுமதி வழங்கு).

mkdir my-ftp
chmod 700 my-ftp 

பின்னர் பின்வரும் கட்டளைகளில் எதாவது ஒன்றை இயக்கி நாம் ftp தளத்தினை இணைக்கலாம்.

curlftpfs ftp://username:password:ftpsite.com ~/my-ftp
(அ)
curlftpts ftpsite.com -o user="username:password" ~/my-ftp
(அ)
curlftpfs ftpsite.com  

எந்த அளவு பாதுகாப்பானது ?

உங்களுக்கு உங்களுடைய ftp வழங்கியின் கடவுச்சொல் செயல்பாடு அளவில் (process level) பிற பயனர்களுக்கு தெரியாமல் இருக்க .netrc கோப்பில் பயனர் பெயர் மற்றும் கடவுச்சொல் ஆகியவற்றை காட்டாமல் பயன்படுத்தலாம்.

~/.netrc எனும் கோப்பினை உங்களது விருப்ப உரைதொகுப்பியில் திறக்கவும். கோப்பு இல்லையேல் புதியதொன்றை உருவாக்கிகொள்ளவும்.

machine ftpsite.com username yourfancyusername password yourhardpassword

கோப்பினில் மேற்கண்டவாறு ftp தளத்தின் பெயர் பயனர் பெயர் மற்றும் கடவுச்சொல் ஆகியவற்றை சேமித்துக்கொள்ளவும்.

கோப்பு ஆனது 600 அனுமதி கொண்டிருப்பதை உறுதிசெய்துகொள்ளவும். உங்களது கடவுச்சொல் வெறும் உரையாக சேமிக்கப்பட்டிருக்கிறது. பிற பயனர்களின் தேவையற்ற அனுகலை இது தடுக்கிறது.

FTP : கோப்பு அனுப்பும் நெறிமுறை plain text ஆக இருப்பதால் அதில் நீங்கள் பெரிய பாதுகாப்பை எதிர்ப்பார்க்க முடியாது உங்களுடைய திறன்பேசியின் hotspot , வீட்டில் இருக்கும் wifi மூலமாக மட்டுமே அணுகுவீர்கள் எனில் நல்லது.

FTPS : கோப்பு அனுப்பும் நெறிமுறை பாதுகாக்கப்பட்டது இதில் மறைப்பாக்கம் (encryption) மூலம் பாதுகாக்கப்படுவதால் ftp யை விட சற்று பாதுகாப்பானது

SFTP : பாதுகாப்பான கோப்பு பரிமாற்ற நெறிமுறை ஆனது sshன் மறைப்பக்க யுக்திகளை பின்பற்றுவதால் மேற்சொன்ன மூன்றில் இது அதிக பாதுகாப்பினை வழங்குகிறது.

live-demo

மேலும் அறிய

https://curlftpfs.sourceforge.net/

https://techworldkb.wordpress.com/2014/07/02/how-to-set-ftp-autologin-with-netrc-file-in-linux/

FUSE- https://medium.com/@goamaral/fuse-filesystem-b44768f27aa2

வினாக்கள் இருப்பின் கமண்ட்ஸ் பகுதியில் கேட்கவும்.

]]> Hariharan Uncategorized https://programmerlife1.wordpress.com/2024/11/08/how-to-create-publish-a-php-package-with-composer-%e0%ae%a4%e0%ae%ae%e0%ae%bf%e0%ae%b4%e0%ae%bf%e0%ae%b2%e0%af%8d/ https://programmerlife1.wordpress.com/2024/11/08/how-to-create-publish-a-php-package-with-composer-%e0%ae%a4%e0%ae%ae%e0%ae%bf%e0%ae%b4%e0%ae%bf%e0%ae%b2%e0%af%8d/ Fri, 08 Nov 2024 17:59:26 GMT அக், 13 2024

பிஹெச்பி பொதிகளை பிஹெச்பி கம்போசர்-உடன் உருவாக்க மற்றும் வெளியிடுவது ஒரு நேரடியான வழிமுறை இந்த வழிமுறையை பின்பற்றினால் நாம் எளிமையாக பிஹெச்பி சமூகத்துடன் நமது நிரல்களை பொதிவடிவத்தில் பகிர்ந்துகொள்ளலாம்.

கம்போசர் – (பிஹெச்பி சார்புகளின் நிர்வாகி) – PHP Dependency Manager

தேவையானவை:

உங்களது கணினியில் பின்வருவற்றை நிறுவி இருப்பது அவசியம்.

• பிஹெச்பி (பதிப்பு 7.4 or அண்மை)
• கம்பொசர் (அண்மை பதிப்பு)
• கிட் (அண்மை பதிப்பு)
• ஒரு கிட் ஹப் கணக்கு
• பேக்கஜிஸ்ட் கணக்கு

படிகள்:

படி 1: நம்முடைய பொதிக்கான ஒரு கோப்புறையை உருவாக்கி கொள்ளவும்.

mkdir open-tamil
cd open-tamil

படி 2: கம்போசர் பொதியை துவக்குதல்

நம் கணினியில் கம்போசர் பொதியை துவக்க பின்வரும் கட்டளையை பயன்படுத்தவும்.

composer init

மேற்கண்ட கட்டளையை பயன்படுத்தும் கட்டளைவரி இடைமுகம் பின்வரும் கேள்விகளை கேட்கும்

Package name: your-username/my-php-package

Description: A sample PHP package

Author: Your Name <your-email@example.com>

Minimum Stability: stable (or leave blank)

Package Type: library

License: MIT

இந்த கேள்விகளுக்கு விடையளித்த பின்பு பிறசார்புகளை கேட்கும் no கொடுக்கவும்.

இறுதியாக composer.json உருவாக்க தூண்டியில் yes கொடுத்து உருவாக்கி கொள்ளவும்.

படி 3 :

composer.json கோப்பு உருவாக்கிய பிறகு அது பின்வருமாறு தோன்றும்

{
    "name": "your-username/my-php-package",
    "description": "A sample PHP package",
    "type": "library",
    "require": {
        "php": ">=7.4"
    },
    "autoload": {
        "psr-4": {
            "MyPackage\\": "src/"
        }
    },
    "authors": [
        {
            "name": "Your Name",
            "email": "your-email@example.com"
        }
    ],
    "license": "MIT"
}

படி 4

பின்னர் உங்களது குறிமுறையை கிட் பயன்படுத்தி கிட்ஹப்பில் பதிவேற்றவும்.

படி 5

குறியீட்டை கம்போசரில் பதிப்பிக்க பேக்கேஜிஸ்டில் உள்நுழையவும். பின்னர் submit பொத்தானை அழுத்தவும்

submit பொத்தானை அழுத்தியவுடன் பொதியை எற்றும் பக்கம் திறக்கப்பட்டு உங்களது கிட்ஹப் கணக்கில் உள்ள பொதுவாக அனுமதியில் இருக்ககூடிய ரெபொசிடரியின் வலைமுகவரியை உள்ளிட்டு சரிபார்க்கும் பொத்தானை அழுத்தி சரிபார்த்துகொள்ளவும்.

குறிப்பு : கம்போசரை பொறுத்தவகையில் பதிப்பிப்பவர் வென்டார் (vendor) என்று குறிப்பிடப்படுவர். நான் hariharan என்ற வென்டார் பெயரை பயன்படுத்தி இரு பொதிகளை பதிப்பித்துள்ளேன்.

புதிய பொதியை சரிபார்த்த பின் பொதியானது பதிப்பிக்க தயராகிவிடும்.

பார்க்க :

https://packagist.org/packages/hariharan/open-tamil

https://packagist.org/packages/hariharan/thirukural

நிறுவி பார்க்க:

composer require hariharan/thirukural

composer require hariharan/open-tamil

]]> Hariharan Uncategorized composer kaniyam tamil https://ravidwivedi.in/posts/kenya-trip/ https://ravidwivedi.in/posts/kenya-trip/ Mon, 04 Nov 2024 19:25:21 GMT In September of this year, I visited Kenya to attend the State of the Map conference. I spent six nights in the capital Nairobi, two nights in Mombasa, and one night on a train. I was very happy with the visa process being smooth and quick. During the conference, I stayed at the Nairobi Transit Hotel with other attendees, with Ibtehal from Bangladesh as my roommate. I found it interesting that the shops around the hotel were grated, presumably to protect against robberies. The hotel guard had to open the lock of the hotel for us to go out at night. Further, I noticed that the banks had three layers of security. Despite this, Ibtehal and I used to hangout at a coffee shop during midnight.

The country lies on the equator, which might give the impression of extremely hot temperatures. However, Nairobi was on the cooler side (10–25 degrees Celsius), and I found myself needing a hoodie, which I bought the next day. It also served as a nice souvenir, as it had an outline of the African map printed on it.

I also bought a Safaricom SIM card for 100 shillings and recharged it with 1000 shillings for 8 GB internet with 5G speeds and 400 minutes talk time.

A visit to Nairobi’s Historic Cricket Ground

On this trip, I got a unique souvenir that can’t be purchased from the market—a cricket jersey worn in an ODI match by a player. The story goes as follows: I was roaming around the market with my friend Benson from Nairobi to buy a Kenyan cricket jersey for myself, but we couldn’t find any. So, Benson had the idea of visiting the Nairobi Gymkhana Club, which used to be Kenya’s main cricket ground. It has hosted some historic matches, including the 2003 World Cup match in which Kenya beat the mighty Sri Lankans and the record for the fastest ODI century by Shahid Afridi in just 37 balls in 1996.

Although entry to the club was exclusively for members, I was warmly welcomed by the staff. Upon reaching the cricket ground, I met some Indian players who played in Kenyan leagues, as well as Lucas Oluoch and Dominic Wesonga, who have represented Kenya in ODIs. When I expressed interest in getting a jersey, Dominic agreed to send me pictures of his jersey. I liked his jersey and collected it from him. I gave him 2000 shillings, an amount suggested by those Indian players.

Giraffe Center in Nairobi

Kenya is known for its safaris and has no shortage of national parks. In fact, Nairobi is the only capital in the world with a national park. I decided not to visit one, as most of them were expensive and offered multi-day tours, and I didn’t want to spend that much time in the wildlife.

Instead, I went to the Giraffe Center in Nairobi with Pragya and Rabina. The ticket cost 1500 Kenyan shillings (1000 Indian rupees). In Kenya, matatus - shared vans, usually decorated with portraits of famous people and play rap songs - are the most popular means of public transport. Reaching the Giraffe Center from our hotel required taking five matatus for 150 shillings and a 2 kilometer walk. The journey back was 90 shillings, suggesting that we didn’t find the most efficient route to get there. At the Giraffe Center, we fed giraffes and took photos.

Train ride from Nairobi to Mombasa

I wanted to visit a place outside of Nairobi. Mombasa being a coastal city and the second-largest in Kenya was a natural choice. It is 500 kilometers from the capital Nairobi. I love trains in general, so I decided to take the SGR train from Nairobi to Mombasa. I tried reserving a seat for myself from home, but found out that the train could only be booked using M-PESA. It is a mobile bank transfer system in Kenya, and I didn’t have an M-PESA account. Therefore, I could not book my ticket in advance. When I was in Kenya, I was helped by Pragya’s friend Mary for booking my ticket. It was a second-class ticket for 1500 shillings (1000 Indian rupees).

If you are a tourist, note that local shops in Kenya can facilitate such an M-PESA transfer in exchange for cash. Your hotel may also be providing such a service, so make sure to check with them.

The train was scheduled to depart from Nairobi at 08:00 hours in the morning and arrive in Mombasa at 14:00 hours. The security check at the station required scanning bags and having them sniffed by sniffer dogs. I also fell victim to a scam by a security official who offered to help me get my ticket printed, only to later ask me to get him some coffee, which I politely declined.

Before boarding the train, I was treated to some stunning views at the Nairobi Terminus station. It was a seating train, but I wished it were a sleeper train, as I was sleep-deprived. The train was neat and clean, with good toilets. It reached Mombasa on time.

Arrival in Mombasa

Mombasa was a bit hotter than Nairobi, with temperatures reaching around 30 degrees Celsius. However, that’s not too hot for me, as I am used to higher temperatures in India. I had booked a hostel in the Old Town and was searching for a hitchhike from the Mombasa Terminus station. After trying for more than half an hour, I took a matatu that dropped me 3 km from my hostel for 200 shillings (140 Indian rupees). I tried to hitchhike again but couldn’t find a ride.

I think I know why I couldn’t get a ride in both the cases. In the first case, the Mombasa Terminus was in an isolated place, so most of the vehicles were taxis or matatus while any noncommercial cars were there to pick up friends and family. If the station were in the middle of the city, there would be many more car/truck drivers passing by, thus increasing my possibilities of getting a ride. In the second case, my hostel was at the end of the city, and nobody was going towards that side. In fact, many drivers told me they would love to give me a ride, but they were going in some other direction.

Finally, I took a tuktuk for 70 shillings to reach my hostel, Tulia Backpackers. It was 11 USD (1400 shillings) for one night. The balcony gave a nice view of the Indian Ocean. The rooms had fans, but there was no air conditioning. Each bed also had mosquito nets. The place was walking distance of the famous Fort Jesus. Mombasa has had more Islamic influence compared to Nairobi and also has many Hindu temples.

Visiting White Sandy Beaches and Hitchhiking

Visiting Nyali beach marked my first time ever at a white sand beach. It was like 10 km from the hostel. The next day, I visited Diani Beach, which was 30 km from the hostel. Going to Diani Beach required crossing a river, for which there’s a free ferry service every few minutes, followed by taking a matatu to Ukunda and then a tuk-tuk. The journey gave me a glimpse of the beautiful countryside of Kenya.

During my return from Diani Beach to the hostel, I was successful in hitchhiking. However, it was only a 4 km ride and not sufficient to reach Ukunda, so I tried to get another ride. When a truck stopped for me, I asked for a ride to Ukunda. Later, I learned that they were going in the same direction as me, so I got off within walking distance from my hostel. The ride was around 30 km. I also learned the difference between a truck ride and a matatu or car ride. For instance, matatus and cars are much faster and cooler due to air conditioning, while trucks tend to be warmer because they lack it. Further, the truck was stopped at many checkpoints by the police for inspections as it carried goods, which is not the case with matatus. Anyways, it was a nice experience, and I am grateful for the ride. I had a nice conversation with the truck drivers about Indian movies and my experiences in Kenya.

Back to Nairobi

I took the SGR train from Mombasa back to Nairobi. This time I took the night train, which departs at 22:00 hours, reaching Nairobi at around 04:00 in the morning. I could not sleep comfortably since the train only had seater seats.

I had booked the Zarita Hotel in Nairobi and had already confirmed if they allowed early morning check-in. Usually, hotels have a fixed checkout time, say 11:00 in the morning, and you are not allowed to stay beyond that regardless of the time you checked in. But this hotel checked me in for 24 hours. Here, I paid in US dollars, and the cost was 12 USD.

Almost Got Stuck in Kenya

Two days before my scheduled flight from Nairobi back to India, I heard the news that the airports in Kenya were closed due to the strikes. Rabina and Pragya had their flight back to Nepal canceled that day, which left them stuck in Nairobi for two additional days. I called Sahil in India and found out during the conversation that the strike was called off in the evening. It was a big relief for me, and I was fortunate to be able to fly back to India without any changes to my plans.

Experience with locals

I had no problems communicating with Kenyans, as everyone I met knew English to an extent that could easily surpass that of big cities in India. Additionally, I learned a few words from Kenya’s most popular local language, Swahili, such as “Asante,” meaning “thank you,” “Jambo” for “hello,” and “Karibu” for “welcome.” Knowing a few words in the local language went a long way.

I am not sure what’s up with haggling in Kenya. It wasn’t easy to bring the price of souvenirs down. I bought a fridge magnet for 200 shillings, which was the quoted price. On the other hand, it was much easier to bargain with taxis/tuktuks/motorbikes.

I stayed at three hotels/hostels in Kenya. None of them had air conditioners. Two of the places were in Nairobi, and they didn’t even have fans in the rooms, while the one in Mombasa had only fans. All of them had good Wi-Fi, except Tulia where the internet overall was a bit shaky.

My experience with the hotel staff was great. For instance, we requested that the Nairobi Transit Hotel cancel the included breakfast in order to reduce the room costs, but later realized that it was not a good idea. The hotel allowed us to revert and even offered one of our missing breakfasts during dinner.

The staff at Tulia Backpackers in Mombasa facilitated the ticket payment for my train from Mombasa to Nairobi. One of the staff members also gave me a lift to the place where I could catch a matatu to Nyali Beach. They even added an extra tea bag to my tea when I requested it to be stronger.

Food

At the Nairobi Transit Hotel, a Spanish omelet with tea was served for breakfast. I noticed that Spanish omelette appeared on the menus of many restaurants, suggesting that it is popular in Kenya. This was my first time having this dish. The milk tea in Kenya, referred to by locals as “white tea,” is lighter than Indian tea (they don’t put a lot of tea leaves).

I also sampled ugali with eggs. In Mombasa, I visited an Indian restaurant called New Chetna and had a buffet thali there twice.

Tips for Exchanging Money

In Kenya, I exchanged my money at forex shops a couple of times. I received good exchange rates for bills larger than 50 USD. For instance, 1 USD on xe.com was 129 shillings, and I got 128.3 shillings per USD (a total of 12,830 shillings) for two 50 USD notes at an exchange in Nairobi, while 127 shillings, which was the highest rate at the banks. On the other hand, for smaller bills such as a one US dollar note, I would have got 125 shillings. A passport was the only document required for the exchange, and they also provided a receipt.

My advice for travelers would be to keep 50 USD or larger bills for exchanging into the local currency while saving the smaller US dollar bills for accommodation, as many hotels and hostels accept payment in US dollars (in addition to Kenyan shillings).

Missed Malindi and Lamu

There were more places on my to-visit list in Kenya. But I simply didn’t have time to cover them, as I don’t like rushing through places, especially in a foreign country where there is a chance of me underestimating the amount of time it takes during transit. I would have liked to visit at least one of Kilifi, Watamu or Malindi beaches. Further, Lamu seemed like a unique place to visit as it has no cars or motorized transport; the only options for transport are boats and donkeys. But I missed Lamu as well.

That’s it for now. Meet you in the next one :)

]]> Ravi Dwivedi https://programmerlife1.wordpress.com/2024/10/31/ubuntu-ssd%e0%ae%b2%e0%af%8d-%e0%ae%89%e0%ae%aa%e0%af%81%e0%ae%a3%e0%af%8d%e0%ae%9f%e0%af%81-20-04-lts-%e0%ae%9f%e0%af%82%e0%ae%af%e0%ae%b2%e0%af%8d-%e0%ae%aa%e0%af%82%e0%ae%9f%e0%af%8d-%e0%ae%ae/ https://programmerlife1.wordpress.com/2024/10/31/ubuntu-ssd%e0%ae%b2%e0%af%8d-%e0%ae%89%e0%ae%aa%e0%af%81%e0%ae%a3%e0%af%8d%e0%ae%9f%e0%af%81-20-04-lts-%e0%ae%9f%e0%af%82%e0%ae%af%e0%ae%b2%e0%af%8d-%e0%ae%aa%e0%af%82%e0%ae%9f%e0%af%8d-%e0%ae%ae/ Thu, 31 Oct 2024 14:18:45 GMT அக் 31, 2024

அண்மையில் நான் டூயல் பூட் முறையில் விண்டோசுடன் நிறுவிய அனுபவத்தினை இந்த பதிவில் காணலாம்.

நான் SSDல் விண்டோஸ் இயங்குதளம் பயன்படுத்திவருகிறேன். என்னுடைய இன்னொரு HDDல் உபுன்டு இயங்குதளம் வைத்திருக்கிறேன். அந்த வன்வட்டு பழுதடையும் தருவாயில் இருப்பதால் SSDல் உபுண்டு இயங்குதளம் நிறுவ தயாரானேன்.

எப்பொழுதும் புதிய இயங்குதளம் நிறுவ தாயராகும் போது காப்பு பிரதி(Backup) எடுத்துவைத்து தயாராகவும்.

குறிப்பு : நான் இயங்குதளம் 3 முறை நிறுவியுள்ள அனுபவத்தில் காப்பு பிரதி எடுக்காமல் தொடங்கினேன். ஆனால் இவ்வாறு செய்வது பரிந்துரைக்கபடவில்லை.

நான் லைவ் USB ventoy எனும் மென்பொருளின் உதவியுடன் தயார் செய்தேன்.

References :

Setting UP ubuntu’s GRUB as primary bootloader using efibootmgr https://superuser.com/questions/1247300/how-to-make-uefi-bios-start-grub-not-windows

]]> Hariharan Linux ubuntu https://mrkaran.dev/posts/using-llm/ https://mrkaran.dev/posts/using-llm/ Wed, 30 Oct 2024 00:00:00 GMT Just yesterday, GitHub announced integrating Claude 3.5 Sonnet with Copilot. Interesting times ahead. In my experience, Claude has been remarkably better than the GPT-4 family of models for programming tasks. I’ve tried a bunch of tools like Cursor, Continue.dev but finally settled with Aider for most of my tasks. In this post, I want to write about my workflow of using Aider when working on small coding tasks.

Aider is an open source Python CLI which supports multiple models, including Claude 3.5 Sonnet. Aider describes itself as “AI pair programming in your terminal”. The tool integrates git quite well in its workflow so it can edit files, create new files, and track all changes via git. In case you want to revert, simply reverting the commit or using the /undo shortcut would do the same.

The tool has multiple modes that serve different purposes:

• /ask: Use it when you simply want to chat with the model about the codebase or explain some pieces of it. This mode won’t touch your files. It’s great for understanding existing code or getting explanations.
• /architect: Use it to discuss a broad overall idea. The model will propose some changes to your files. You can further chat and tune it to your preferences.
• /code: This will directly edit your files and commit them.

My typical workflow involves running Aider in a terminal while keeping VSCode open for manual code review. I often use the --no-auto-commits flag to view the diffs before committing. Despite advances in LLM technology, I believe they haven’t yet reached the stage where they can fully understand your team’s coding style guides, and I prefer not to have a certain style forced upon me. Manually tweaking portions of AI-generated functions still proves helpful and saves considerable time.

To begin, aider --sonnet would open the interactive window where you can begin writing prompts.

To add context, you need to add files using commands like /add main.py. What makes Aider powerful is its control over the LLM context - you can /add or /drop source code, or even /reset to drop all files and start with a fresh context. This granular control helps manage the context window effectively.

A really cool thing about it is that it gives an approximate idea of the number of tokens (cost) associated with each prompt. I find it useful to remove unnecessary files from the context window, which not only helps in getting sharper, more accurate responses but also helps with the costs. There’s a nice /tokens command which will show the cost of sending each file added in context with the prompt.

I find the Aider + Claude 3.5 combo works really well when you have a narrow-scoped, well-defined task. For example, this is the prompt I used on a codebase I was working on:

Theme preference is not preserved when reloading pages or navigating to new pages. We should store this setting in localStorage. Please implement using standard best practices.

Under the hood, Aider uses tree-sitter to improve code generation and provide rich context about your codebase. Tree-sitter parses your code into an Abstract Syntax Tree (AST), which helps Aider understand the structure and relationships in your code. Unlike simpler tools that might just grep through your codebase, tree-sitter understands the actual syntax of your programming language.

• It can identify function definitions, class declarations, variable scopes, and their relationships
• It extracts full function signatures and type information
• It builds a dependency graph showing how different parts of your code relate to each other
• It helps rank the importance of different code sections based on how often they’re referenced

This means when you’re working on a task, Aider isn’t just blindly sending your entire codebase to the LLM. Instead, it creates an optimized “repository map” that fits within your token budget (default is 1k tokens, adjustable via --map-tokens). This map focuses on the most relevant pieces of your code, making sure the LLM understands the context without wasting tokens.

Aider’s approach to AI pair programming feels natural and productive. Here are some example prompts where it helped me build stuff in less than a minute:

Modify fetch method in store/store.go to filter out expired entries

Write a k6 load test script to benchmark the POST /submit endpoint and simulate real-world traffic patterns

Create a Makefile, Dockerfile, goreleaser.yml for my Go binary. Target platforms: arm64 and amd64

I prefer to invoke aider with a few extra flags:

aider --no-auto-commits --cache-prompts --cache-keepalive-pings 12 --no-suggest-shell-commands

Make sure to go through the Tips page to effectively try out Aider on your existing projects.

Fin!

]]> Karan Sharma https://aryak.me/blog/08-fossmeetup-livestream.html https://aryak.me/blog/08-fossmeetup-livestream.html Thu, 17 Oct 2024 12:39:45 GMT , and enable the new destination. By doing this, all new streams will be simultaneously streamed to YouTube as well. Making a clip for each talk After the live stream has ended, you might want to separate the stream into smaller chunks, with a separate video for each of the talks. This can be done using the clips feature of OSP. Once the stream ends, the stream will be converted into a recording and uploaded on to the channel from where it was being streamed. Open the video, and in it the cogweel, from where you can select the create clip option. Using the slider, move to the desired start and end point, give it a valid description and title, and then create the clip. A video is attached for reference: Video OBS Setup Requirements: OBS Installation, with the default browser plugin enabled (the plugin is not enabled on debian as of writing FYI, so use the flatpak version if on Debian) An X11 environment (if under wayland, run env -u WAYLAND_DISPLAY obs Run setup wizard, optimize for streaming, and use 1920x1080 as the base resolution. For stream key, enter the one from your service provider (if you are using OSP, consult the next section) Disable MIC/AUX, and keep Desktop Audio enabled In OBS, add a window source (Labelled XComposite Window Capture on linux) for scrcpy In OBS, add a browser source, and make the url point towards the screensy link that you copied initially. If you are using VDO.ninja instead, the url will be the one from the “capture group scene” from the VDO.ninja setup. At the end, it should look something like this: Start streaming!]]> I volunteer for FOSS United Mumbai, and we organize meetups related to FOSS every month.

We have often done some hacky jugaad to stream our meetups, but due to their last minute nature and lack of proper resources, they have often been low quality or failed entirely.

With the on-ground experience I have gained as the live streaming lead at IndiaFOSS 2024 and 2023, I wanted to put these ideas into a blog-post format, so this could help out the other chapters of FOSS United, and potentially non-FOSS United events too!

My guide is primarily divided into four parts:

• Scrcpy, which deals with getting the video of the meetup by (ab)using the camera of your phone
• VDO.ninja, which retrieves a high quality, low-latency stream of the speaker’s screen in a user-friendly manner
• OBS, which combines the outputs of both Scrcpy and VDO.ninja together with a nice template, to send to the streaming server (OSP, or perhaps YouTube)
• OSP (FOSS United specific), which is the web UI which we stream to, and use to make clips out of the live stream

A large part of the guide assumes you use linux, but it should work with MacOS. PS: if you are hosting a FOSS Meetup, streaming from a Mac/Windows Box doesn’t make much sense either way :-)

Scrcpy Setup

Requirements:

• Laptop with Linux/MacOS (only tested with linux though)
• Scrcpy 2.x
• Android 12+ phone

On the phone:

• Enable Developer Options by tapping Build Number 7 times in the Software Information section of About Phone in the Settings app.
• Go to the newly enabled developer options menu, allow USB Debugging.
• Connect phone to the laptop via USB, and authorize the Laptop
  • If there is no option asking you to authorize, there should be a silent notification in your notif tray regarding USB preferences. Click that and select ADB (if not available, chose file transfer - that should be an equivalent option)
  • It might also be that the cable is old/damaged, and does not support data transfer. If previous option didn’t yield any results, try using a different cable

On the laptop:

• Install ADB
• Install Scrcpy (Linux | MacOS | Windows)
• Run this command: /usr/local/bin/scrcpy --video-source=camera --camera-id=0 --audio-source=mic --orientation=90
  • Sometimes the ideal resolution and framerate will not be the perfect 16:9 one that is required. In that case, add the following argument: --camera-size=1920x1080 --camera-fps=60
  • If you want to add any custom settings, consult this other blogpost of mine or the scrcpy docs
  • If you want a landscape camera output, remove the –orientation flag
  • Setting the camera ID to 0 should use the back camera by default, but if front camera needs to be used, set the ID to 1
• This should open a window with a full-sized preview of the camera:

Screensy / Screenshare Setup

I would recommend you use this instead of VDO.ninja, ONLY IF BOTH DEVICES ARE ON SAME NETWORK/WIFI.

The Setup

• On the speaker’s laptop, open https://screensy.marijn.it. Start screen sharing and note down the URL (making sure that the text that follows the # is included)

VDO.Ninja / Screenshare Setup

NOTE: YOU ARE BETTER OFF USING SCREENSY INSTEAD - UNLESS YOU CANNOT GET BOTH LAPTOPS ON THE SAME NETWORK.

Requirements:

• A browser which supports screenshare on the speaker’s PC, any browser on the streaming PC

The Setup:

• On streaming PC, open vdo.ninja, and create a room. Give a valid room name, and set the option select the “The guests can see the director, but not other guests’ videos” option, and then enter the control centre
• From the control centre, copy the link to invite a guest, and ask the next speaker to open it on his laptop.
• On the speaker’s laptop, select screenshare with room, and then in the settings cogweel, make it use highest quality.
• Back on the streaming PC in the control centre, press highlight on the newly appeared preview, and then copy link of “capture a group scene”. Keep the link safe, its needed in the next OBS setup

Open Streaming Platform Setup (specific to FOSS United)

On https://stream.fossunited.org, each city chapter/foss club can request an account for streaming and uploading their talks. Login to the account that is provided to you, navigate to My Channels and create a new channel.

Add the specific details like default title, description, profile picture etc., and then copy the auto-generated stream key.

If you want to stream simultaneously to youtube, you can add a new RTMP Restream Destination in this format: rtmp://a.rtmp.youtube.com/live2/<STREAM KEY>, and enable the new destination. By doing this, all new streams will be simultaneously streamed to YouTube as well.

Making a clip for each talk

After the live stream has ended, you might want to separate the stream into smaller chunks, with a separate video for each of the talks. This can be done using the clips feature of OSP.

Once the stream ends, the stream will be converted into a recording and uploaded on to the channel from where it was being streamed.

Open the video, and in it the cogweel, from where you can select the create clip option. Using the slider, move to the desired start and end point, give it a valid description and title, and then create the clip. A video is attached for reference:

OBS Setup

Requirements:

• OBS Installation, with the default browser plugin enabled (the plugin is not enabled on debian as of writing FYI, so use the flatpak version if on Debian)

• An X11 environment (if under wayland, run env -u WAYLAND_DISPLAY obs

• Run setup wizard, optimize for streaming, and use 1920x1080 as the base resolution. For stream key, enter the one from your service provider (if you are using OSP, consult the next section)

• Disable MIC/AUX, and keep Desktop Audio enabled

• In OBS, add a window source (Labelled XComposite Window Capture on linux) for scrcpy

• In OBS, add a browser source, and make the url point towards the screensy link that you copied initially. If you are using VDO.ninja instead, the url will be the one from the “capture group scene” from the VDO.ninja setup.

• At the end, it should look something like this:

• Start streaming!

]]> arya@projectsegfau.lt (Arya Kiran) 2024/10/17/4 https://www.learningfromdata.zingg.ai/p/celebrating-3-years-of-open-source https://www.learningfromdata.zingg.ai/p/celebrating-3-years-of-open-source Wed, 02 Oct 2024 06:05:00 GMT

“The Data Weaver’s Tale”
In fields of data, vast and wide,
Where records dance and entities hide,
A weaver works with threads so fine,
To link and match and intertwine.

Deduplication clears the way,
As fuzzy matching comes to play.
Record linkage, strong and true,
Brings scattered pieces into view.

Entity linking, precise and keen,
Connects the dots once unforeseen.
The knowledge graph, a tapestry,
Reveals hidden identity.

With AI’s wisdom as our guide,
We stitch and resolve, side by side.
In this grand quest for truth and light,
Entity resolution shines so bright.

When I open sourced Zingg three years ago, Entity Resolution felt like an esoteric problem. While I had personally experienced the issue of unharmonized records, it was hard to say if I was an outlier or if this was a much more common problem. Sure, master data management systems have existed since the beginning of the data industry. Or even before it. Customer data platforms have also had their share of the spotlight. Yet, for all of the modern data stack maps out there, identity and entity resolution have been largely omitted.

For someone who had burnt their life’s savings in working on Zingg, I was treading unknown territory. The encouragement I had received from people who understood the problem truly meant a lot to me. Yet, the way forward was unclear. Starting and building a company is terribly hard. It was tempting to think of a cushioned job which would save me from a lot of hassles, guarantee a handsome package and possibly some other interesting problems.

However, I could not bring myself to do anything else - entity resolution felt like a problem I could not forget. If I could help a few practitioners like myself who had struggled with entity resolution, I would be satisfied. If I think back, it is likely that there was a builder bias in my thinking. Working on Zingg is fun! The challenges we see are not common place. They force you to think, they push you to the drawing board and scratch your head again and again in the quest to find a solution. It is wonderful to apply tools and techniques learnt over multiple years, yet see oneself falling short and researching and learning more and more. Working on Zingg feels like an endless maze of hard computational puzzles thrown our way. As a professional, one could not ask for more!

Also, somewhere deep down, I was not too worried about the selling bit. I was convinced that it was worth a try. One big reason was that the existing tooling in entity resolution is outdated. MDM and CDP tools aim to do too much, and the core problem that they try to solve - entity resolution, is something Zingg specializes in. Zingg’s identity resolution capabilities make it a drop in replacement for an MDM system and a key component in composable CDP stacks. Zingg can work with large datasets and resolve different kinds of entities with ease. We are privacy first, with no data ever leaving customer premises. Our warehouse/lakehouse native approach enables an elegant data architecture, letting people leverage their existing investment into data tooling and running Zingg AI as part of their data pipeline. Without any specialized AI skills, one can straight off use Zingg results in marketing attribution, risk or personalisation models. Or pipe Zingg resolved entities to the RAG of an LLM model or to a knowledge graph.

I have surely made my share of mistakes in life, but turns out I was right on betting wholeheartedly on Zingg. Adoption by open source users has been motivating. Due to on premise and multicloud environments along with the proliferation of SAAS tools, there is a growing need for trusted and enriched entity data. The race to deploy large language models, enterprise knowledge graphs and RAG based systems is pushing the demand for entity resolution even more. An enterprise data leader recently told me that Customer 360 is among the top 3 problems their team worries about. We are repeatedly hearing single customer view, SCV, Customer 360, C360 and variants in sales conversations. A very healthy and growing revenue from Zingg Enterprise is giving us the lever to go deeper into the problem, innovate even further and build simple yet delightful experiences for end users.

We are onto something special, and it is an experience worth having!

This would not have been possible without some really solid support and I am extremely grateful for that.

A big thank you to each of our users for trying Zingg, mentioning us to friends and coworkers and for joining our Slack. We hope that Zingg will continue to help you in your data journey. Special thanks to our investors and paying customers of Zingg Enterprise. We could not have reached here without you. Thank you for your early confidence and bet on us. We have built a lot of valuable features like ZINGG_ID, incremental flows, and determinstic matching based on customer asks and our understanding of customer needs. The product is so much richer due to your feedback and feature asks.

A lot of ground has been covered in the last 3 years, and a lot more remains. In the coming months, we plan to grow the team to build more cool stuff we have been aching to, and continue to dabble in interesting problems at the intersection of technology and company building. Wish us luck!

Thank you for reading, and if this story resonates with you, do say hello!

]]> Sonal Goyal https://nadh.in/blog/decentralised-open-indexes/ https://nadh.in/blog/decentralised-open-indexes/ Wed, 02 Oct 2024 00:00:00 GMT TLDR; A conceptual and technical framework for resource discovery on the WWW using decentralised, open, machine-readable indexes as the building block, free of eroding quality and gatekeeping by BigSearch™ and BigPlatform™, whose goals are not quality, but revenue.

]]> Kailash Nadh https://ravidwivedi.in/posts/sotm-2024/ https://ravidwivedi.in/posts/sotm-2024/ Tue, 01 Oct 2024 14:05:30 GMT Last month, I traveled to Kenya to attend a conference called State of the Map 2024 (“SotM” for short), which is an annual meetup of OpenStreetMap contributors from all over the world. It was held at the University of Nairobi Towers in Nairobi, from the 6th to the 8th of September.

I have been contributing to OpenStreetMap for the last three years, and this conference seemed like a great opportunity to network with others in the community. As soon as I came across the travel grant announcement, I jumped in and filled the form immediately. I was elated when I was selected for the grant and couldn’t wait to attend. The grant had an upper limit of €1200 and covered food, accommodation, travel and miscellaneous expenses such as visa fee.

Pre-travel tasks included obtaining Kenya’s eTA and getting a yellow fever vaccine. Before the conference, Mikko from the Humanitarian OpenStreetMap Team introduced me to Rabina and Pragya from Nepal, Ibtehal from Bangladesh, and Sajeevini from Sri Lanka. We all booked the Nairobi Transit Hotel, which was within walking distance of the conference venue. Pragya, Rabina, and I traveled together from Delhi to Nairobi, while Ibtehal was my roommate in the hotel.

The venue, University of Nairobi Towers, was a tall building and the conference was held on the fourth, fifth and sixth floors. The open area on the fifth floor of the building had a nice view of Nairobi’s skyline and was a perfect spot for taking pictures. Interestingly, the university had a wing dedicated to Mahatma Gandhi, who is regarded in India as the Father of the Nation.

The diversity of the participants was mind-blowing, with people coming from a whopping 54 countries. I was surprised to notice that I was the only participant traveling from India, despite India having a large OpenStreetMap community. That said, there were two other Indian participants who traveled from other countries. I finally got to meet Arnalie (from the Phillipines) and Letwin (from Zimbabwe), both of whom I had only met online before. I had met Anisa (from Albania) earlier during DebConf 2023. But I missed Mikko and Honey from the Humanitarian OpenStreetMap Team, whom I knew from the Open Mapping Guru program.

I learned about the extent of OSM use through Pragya and Rabina’s talk; about the logistics of running the OSM Board, in the OSMF (OpenStreetMap Foundation) session; about the Youth Mappers from Sajeevini, about the OSM activities in Malawi from Priscilla Kapolo, and about mapping in Zimbabwe from Letwin. However, I missed Ibtehal’s lightning session. The ratio of women speakers and participants at the conference was impressive, and I hope we can get such gender representation in our Delhi/NCR mapping parties.

Outside of talks, the conference also had lunch and snack breaks, giving ample time for networking with others. In the food department, there were many options for a lacto-ovo vegetarian like myself, including potatoes, rice, beans, chips etc. I found out that the milk tea in Kenya (referred to as “white tea”) is usually not as strong compared to India, so I switched to coffee (which is also called “white coffee” when taken with milk). The food wasn’t spicy, but I can’t complain :) Fruit juices served as a nice addition to lunch.

At the end of the second day of the conference, there was a surprise in store for us — a bus ride to the Bao Box restaurant. The ride gave us the experience of a typical Kenyan matatu (privately-owned minibuses used as share taxis), complete with loud rap music. I remember one of the songs being Kraff’s Nursery Rhymes. That day, I was wearing an original Kenyan cricket jersey - one that belonged to Dominic Wesonga, who represented Kenya in four ODIs. This confused Priscilla Kapolo, who asked if I was from Kenya! Anyway, while it served as a good conversation starter, it didn’t attract as much attention as I expected :) I had some pizza and chips there, and later some drinks with Ibtehal. After the party, Piyush went with us to our hotel and we played a few games of UNO.

I am grateful to the organizers Laura and Dorothea for introducing me to Nikhil when I was searching for a companion for my post-conference trip. Nikhil was one of the aforementioned Indian participants, and a wildlife lover. We had some nice conversations; he wanted to go to the Masai Maara Natural Reserve, but it was too expensive for me. In addition, all the safaris were multi-day affairs, and I wasn’t keen on being around wildlife for that long. Eventually I chose to go my own way, exploring the coastal side and visiting Mombasa.

While most of the work regarding the conference was done using free software (including the reimbursement form and Mastodon announcements), I was disappointed by the use of WhatsApp for coordination with the participants. I don’t use WhatsApp and so was left out. WhatsApp is proprietary software (they do not provide the source code) and users don’t control it. It is common to highlight that OpenStreetMap is controlled by users and the community, rather than a company - this should apply to WhatsApp as well.

My suggestion is to use XMPP, which shares similar principles with OpenStreetMap, as it is privacy-respecting, controlled by users, and powered by free software. I understand the concern that there might not be many participants using XMPP already. Although it is a good idea to onboard people to free software like XMPP, we can also create a Matrix group, and bridge it with both the XMPP group and the Telegram group. In fact, using Matrix and bridging it with Telegram is how I communicated with the South Asian participants. While it’s not ideal - as Telegram’s servers are proprietary and centralized - but it’s certainly much better than creating a WhatsApp-only group. The setup can be bridged with IRC as well. On the other hand, self-hosted mailing lists for participants is also a good idea.

Finally, I would like to thank SotM for the generous grant, enabling me to attend this conference, meet the diverse community behind OSM and visit the beautiful country of Kenya. Stay tuned for the blog post on Kenya trip.

Thanks to Sahilister, Contrapunctus, Snehal and Badri for reviewing the draft of this blog post before publishing.

]]> Ravi Dwivedi https://www.prashanthudupa.com/the-end-of-suffering/ https://www.prashanthudupa.com/the-end-of-suffering/ Tue, 01 Oct 2024 06:10:22 GMT Prashanth N Udupa Insight https://mrkaran.dev/posts/setting-outline/ https://mrkaran.dev/posts/setting-outline/ Fri, 20 Sep 2024 00:00:00 GMT I recently discovered Outline a collaborative knowledge base. I wanted to self-host it on my server, but the mandatory auth provider requirement was off-putting. My server is on a private encrypted network (Tailscale) that only my approved devices in the tailnet can access, so I don’t really need authentication for my personal single-use apps. I found a few guides using Authelia/Keycloak, but these are heavy-duty applications that would consume a lot of resources (DBs, caches, proxies, and whatnot) just to have an OIDC provider for Outline.

There had to be a simpler way, right? Enter Dex. As recommended by my friend and colleague Chinmay, it turned out to be quite easy.

Here’s the full docker-compose.yml setup you need to get Outline up and running on your local instance!

services:

  outline:
    image: docker.getoutline.com/outlinewiki/outline:latest
    env_file: ./docker.env
    ports:
      - "3000:3000"
    volumes:
      - storage-data:/var/lib/outline/data
    depends_on:
      - postgres
      - redis
    environment:
      PGSSLMODE: disable

  redis:
    image: redis
    env_file: ./docker.env
    ports:
      - "6379:6379"
    healthcheck:
      test: [ "CMD", "redis-cli", "ping" ]
      interval: 10s
      timeout: 30s
      retries: 3

  postgres:
    image: postgres
    env_file: ./docker.env
    ports:
      - "5432:5432"
    volumes:
      - database-data:/var/lib/postgresql/data
    healthcheck:
      test: [ "CMD", "pg_isready", "-d", "outline", "-U", "user" ]
      interval: 30s
      timeout: 20s
      retries: 3
    environment:
      POSTGRES_USER: 'user'
      POSTGRES_PASSWORD: 'pass'
      POSTGRES_DB: 'outline'

  dex:
    image: dexidp/dex:v2.35.3
    ports:
      - "5556:5556"
    volumes:
      - ./dex:/etc/dex
    command: [ "dex", "serve", "/etc/dex/config.yaml" ]

volumes:
  storage-data:
  database-data:

You’ll need to add the following env variables as well

NODE_ENV=production
SECRET_KEY=your-key
UTILS_SECRET=your-key
DATABASE_URL=postgres://user:pass@postgres:5432/outline
PGSSLMODE=disable
REDIS_URL=redis://redis:6379
URL=http://localhost:3000
PORT=3000
FILE_STORAGE=local
FILE_STORAGE_LOCAL_ROOT_DIR=/var/lib/outline/data
FILE_STORAGE_UPLOAD_MAX_SIZE=262144000
OIDC_CLIENT_ID=outline
OIDC_CLIENT_SECRET=outline-secret
OIDC_AUTH_URI=http://localhost:5556/dex/auth
OIDC_TOKEN_URI=http://dex:5556/dex/token
OIDC_USERINFO_URI=http://dex:5556/dex/userinfo
OIDC_USERNAME_CLAIM=preferred_username
OIDC_DISPLAY_NAME=Dex
OIDC_SCOPES=openid profile email
FORCE_HTTPS=false
ENABLE_UPDATES=true
WEB_CONCURRENCY=1
DEBUG=http
LOG_LEVEL=info

And finally, to configure Dex, we need the following config:

issuer: http://localhost:5556/dex

storage:
  type: sqlite3
  config:
    file: /var/dex/dex.db

web:
  http: 0.0.0.0:5556

staticClients:
  - id: outline
    redirectURIs:
      - "http://localhost:3000/auth/oidc.callback"
    name: "Outline"
    secret: outline-secret

oauth2:
  skipApprovalScreen: true

enablePasswordDB: true

staticPasswords:
  - email: "admin@example.com"
    hash: "$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"
    username: "admin"
    userID: "08a8684b-db88-4b73-90a9-3cd1661f5466"

Voilà! With docker compose up, you’ll have an Outline server ready to go. You can log in using the admin user.

]]> Karan Sharma https://www.evalapply.org/posts/clojure-web-app-from-scratch/index.html https://www.evalapply.org/posts/clojure-web-app-from-scratch/index.html Sat, 24 Aug 2024 00:00:00 GMT Aditya Athalye clojure web_development websites functional_programming software_design architecture howto whyto https://www.learningfromdata.zingg.ai/p/zingg-incremental-flow https://www.learningfromdata.zingg.ai/p/zingg-incremental-flow Thu, 22 Aug 2024 17:20:16 GMT As a startup founder and someone who actively follows companies doing interesting work, I often wonder about the levers behind a viable and thriving business. One thing that comes up repeatedly is Growth.

From the point of view of data teams in a fast growing business, what really does growth amount to? How does it manifest in the day to day data collection and processing?

If we dissect the key activities happening across businesses during the growth phase, there are a few commonalities that stand out. New customers get acquired at a fast pace. Existing customers buy more products and services. They also buy from additional channels, like a website, if they earlier only bought from the brick and mortar store. The business keeps in touch with existing customers and makes sure it has the ways and means to contact them with offers and product updates. Customers inform the business when they move or change jobs. Strategic mergers and acquisitons happen. Additional procurement of products and services happens to fuel the growth and new suppliers get onboarded. New product lines are introduced and additional customers onboarded.

Thus, data about core business entities - customers, suppliers, products and parts starts changing rapidly. New entity records get added, existing ones get updated. Some of the records are deleted.

A small detour before we go any further. In my last post, I discussed about the cross reference and persistent ZINGG_ID. Let us recap a bit. When we build our customer journeys, segment users and attribute acquisition to campaigns and channels, we need a consistent view of the customer data. We want to associate an identifier that inherently represents the entity. And we want to use this identifier in all our reporting and analytics, as well as operational systems. If a customer gets in touch with us at the call centre, we want to use the ZINGG_ID to get the context of their purchases and their relationship with us.

Coming back on the current topic. There is immense business value in having a persistent and continuously updated identity graph for each of the downstream usecases like marketing attribution, personalisation, life time value, fraud and risk.

However, as we saw earlier, in a growing business, records are no longer static. They are getting added and deleted and updated. So, how does all this change affect identity resolution? New records would start matching with existing ones. Updated records may start matching with other records they did not match with earlier. They may stop matching with records they matched up earlier. Hence clusters of resolved identities are evolving with the data. They are merging or splitting and new ones are created all the time.

Let us dig deeper how that looks by going over some examples.

In the following dataset, the records with ID rec-1020-org-incr and rec-1023-org-incr have been updated during business operations and continue to exhibit strong similarity with their original matches. Two new records with ID rec-10000-org-new got added, matched with no other existing record but with each other.

With me so far? Now, here comes an even more interesting bit! In the following dataset, rec-1022-org and rec-1022-dup-4 had matched originally.

In an update at a later date, rec-1022-dup-4 got altered and these records now stop matching with each other. This usually happens in the case of wrong data capture in an original record.

Thus, when a record is updated yet it is close enough to the original cluster values, we ought to keep the cluster relationship. In the case of adding a new record that does not match any existing ones, a new cluster should get created. When a new record shows up that matches an existing cluster, the new record has to get attached to the cluster. When a new record that matches two clusters is passed, the clusters need to get merged. If we make edits to a record so it no longer matches with its existing cluster and then reverted those edits, the models should correctly track the updated and reversed entry, with the record remaining in the correct cluster throughout the incremental runs. There are myriad possible scenarios of clusters splitting and merging with each other. Matching new and updated against existing records gets extremely complex before we realise.

Technically, we could run a full match with Zingg Community version on the updated dataset and get the new clusters.

But.

If we run the entire matching process again, how do we preserve the persistent ZINGG_ID for the records? How do we tie back the newly generated ZINGG_IDs on the updated set to the original ZINGG_IDs? Especially when a new cluster will now have multiple older ZINGG_IDs, or when a new cluster may have some old different ZINGG_IDs and some fresh ones. How do we choose the surviving ZINGG_ID? On top of that, running full matching loads on growing datasets is costly in terms of time taken and compute.

Here is how we handle this in Zingg Enterprise.

Clusters automatically merge, unmerge and new clusters with ZINGG_IDs are created based on new and updated values while only matching the updated records.

Which means that

The identity graph stays updated even when the data changes.

When I had open sourced Zingg, I felt that it was my life’s best work and wondered if I would ever work on anything more challenging. While building the incremental update, I realized that this functionality is way way tougher to build than everything else I had built so far. Magnitudes tougher than the active learning aspects of Zingg, or the blocking model. There are tons of algorithmic and performance aspects to consider. Even chalking out the test strategy and test data proved to be quite a challenge.

After intensive testing on internally generated datasets, we released the runIncremental phase to our early customers. We knew our test data could not mimic every possible scenario, and though we were confident of the algorithms, we suspected edge cases which we had not envisaged. We requested our customers to test the incremental flows on their data and to report any inconsistencies in matching or cluster assignment. We were thrilled to get an extremely positive response!

Here is an excerpt from one of their test reports:

Each test scenario was carefully constructed to challenge the model and verify the expected behavior. The results were consistently positive, demonstrating the model's precision and adaptability in every test case. Its ability to accurately handle minor data modifications and correctly process entirely new, atypical entries was particularly noteworthy. Equally impressive was its capability to manage entries that aligned with several clusters without compromising accuracy and its efficiency in reverting changes to their original form.

The success of these tests is an essential step on the way to releasing the incremental Zingg models in production.

We are super proud to have come so far, and happy to roll it to all our customers. If the idea of an identity graph on your own data within your own data lake and warehouse appeals to you, why not hit reply and let us grab a coffee? :-)

]]> Sonal Goyal https://ravidwivedi.in/posts/austrian-visa-refusal-jan-2024/ https://ravidwivedi.in/posts/austrian-visa-refusal-jan-2024/ Sun, 11 Aug 2024 06:29:04 GMT Vienna - the capital of Austria - is one of the most visited cities in the world, popular for its rich history, gardens, and cafes, along with well-known artists like Beethoven, Mozart, Gödel, and Freud. It has also been consistently ranked as the most livable city in the world.

For these reasons, I was elated when my friend Snehal invited me last year to visit Vienna for a few days. We included Christmas and New Year’s Eve in my itinerary due to the city’s popular Christmas markets and lively events. The festive season also ensured that Snehal had some days off for sightseeing.

Indians require a visa to visit Austria. Since the travel dates were near, I rushed to book an appointment online with VFS Global in Delhi, and quickly arranged the required documents. However, at VFS, I found out that I had applied in the wrong appointment category (tourist), which depends on the purpose of the visit, and that my travel dates do not allow enough time for visa authorities to make a decision. Apparently, even if you plan to stay only for a part of the trip with the host, you need to apply under the category “Visiting Friends and Family”.

Thus, I had to book another appointment under this category, and took the opportunity to shift my travel dates to allow at least 15 business days for the visa application to be processed, removing Christmas and New Year’s Eve from my itinerary.

The process went smoothly, and my visa application was submitted by VFS. For reference, here’s a list of documents I submitted -

• VFS appointment letter

• Duly-filled visa application form

• Original passport

• Copy of passport

• 1 photograph

• My 6 months bank account statement

• Cover letter

• Consent form (that visa processing will take up to 15 business days)

• Snehal’s job contract

• My work contract

• Rent contract of Snehal

• Residence permit of Snehal

• A copy of Snehal’s passport

• Invitation letter from Snehal

• Return flight ticket reservations

• Travel insurance for the intended travel dates

The following charges were collected from me.

Service Description	Amount (Indian Rupees)
Cash Handling Charge - SAC Code: (SAC:998599)	0
VFS Fee - India - SAC Code: (SAC:998599)	1,820
VISA Fee - India - SAC Code:	7,280
Convenience Fee - SAC Code: (SAC:998599)	182
Courier Service - SAC Code: (SAC:998599)	728
Courier Assurance - SAC Code: (SAC:998599)	182
Total	10,192

I later learned that the courier charges (728 INR) and the courier assurance charges (182 INR) mentioned above were optional. However, VFS didn’t ask whether I wanted to include them. When the emabssy is done processing your application, it will send your passport back to VFS, from where you can either collect it yourself or get it couriered back home, which requires you to pay courier charges. However, courier assurance charges do not add any value as VFS cannot “assure” anything about courier and I suggest you get them removed.

My visa application was submitted on the 21st of December 2023. A few days later, on the 29th of December 2023, I received an email from the Austrian embassy asking me to submit an additional document -

Subject: AUSTRIAN VISA APPLICATION - AMENDMENT REQUEST: Ravi Dwivedi VIS 4331

Dear Applicant,

On 22.12.2023 your application for Visa C was registered at the Embassy. You are requested to kindly send the scanned copies of the following documents via email to the Embassy or submit the documents at the nearest VFS centre, for further processing of your application:

• Kindly submit Electronic letter of guarantee “EVE- Elektronische Verpflichtungserklärung” obtained from the “Fremdenpolizeibehörde” of the sponsor’s district in Austria. Once your host company/inviting company has obtained the EVE, please share the reference number (starting from DEL_____) received from the authorities, with the Embassy.

I misunderstood the required document (the EVE) to be a scanned copy of the letter of guarantee form signed by Snehal, and responded by attaching it.

Upon researching, Snehal determined that the document is an electronic letter of guarantee, and is supposed to be obtained at a local police station in Vienna. He visited a police station the next day and had a hard time conversing due to the language barrier (German is the common language in Austria, whereas Snehal speaks English). That day was a weekend, so he took an appointment for Monday, but in the meantime the embassy had finished processing my visa.

My visa was denied, and the refusal letter stated:

The Austrian embassy in Delhi examined your application; the visa has been refused.

The decision is based on the following reason(s):

• The information submitted regarding the justification for the purpose and conditions of the intended stay was not reliable.

• There are reasonable doubts as to your intention to leave the territory of the Member States before the expiry of the visa.

Other remarks:

You have been given an amendment request, which you have failed to fulfil, or have only fulfilled inadequately, within the deadline set.

You are a first-time traveller. The social and economic roots with the home country are not evident. The return from Schengen territory does therefore not seem to be certain.

I could have reapplied after obtaining the EVE, but I didn’t because I found the following line

The social and economic roots with the home country are not evident.

offensive for someone who was born and raised in India, got the impression that the absence of electronic guarantee letter was not the only reason behind the refusal, had already wasted 12,000 INR on this application, and my friend’s stay in Austria was uncertain after January. In fact, my friend soon returned to India.

To summarize -

1. If you are visiting a host, then the category of appointment at VFS must be “Visiting Friends and Family” rather than “Tourist”.
2. VFS charged me for courier assurance, which is an optional service. Make sure to get these removed from your bill.
3. Neither my travel agent nor the VFS application center mentioned the EVE.
4. While the required documents list from the VFS website does mention it in point 6, it leads to a dead link.
5. Snehal informed me that a mere two months ago, his wife’s visa was approved without an EVE. This hints at inconsistency in processing of applications, even those under identical categories.

Such incidents are a waste of time and money for applicants, and an embarrassment to VFS and the Austrian visa authorities. I suggest that the Austrian visa authorities fix that URL, and provide instructions for hosts to obtain the EVE.

Credits to Snehal and Contrapunctus for editing, Badri for proofreading.

]]> Ravi Dwivedi https://kcsrk.info/ocaml/offcputime/bpfcc/2024/07/24/offcputime-analysis/ https://kcsrk.info/ocaml/offcputime/bpfcc/2024/07/24/offcputime-analysis/ Wed, 24 Jul 2024 09:48:00 GMT offcputime.out The command instruments the watches for 10s and the writes out the stack traces corresponding to blocking calls in offcputime.out. We use a large stack storage size argument so as to not lose stack traces. Otherwise, you will see many [Missing User Stack] errors in the back traces. Caveats offcputime-bpfcc must run longer than the program being instrumented by a few seconds so that the function symbols are resolved. Otherwise you may see [unknown] in the backtrace for function names. Oddities I still see an order of magnitude difference between the maximum pauses observed using offcputime-bpfcc and olly trace. Something is off. Other links https://www.pingcap.com/blog/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance/]]> Off-CPU analysis is where the program behavior when it is not running is recorded and analysed. See Brendan Gregg’s eBPF based off-CPU analysis. While on-CPU performance monitoring tools such as perf give you an idea of where the program is actively spending its time, they won’t tell you where the program is spending time blocked waiting for an action. Off-CPU analysis reveals information about where the program is spending time passively.

Installation

Install the tools from https://github.com/iovisor/bcc/.

Enabling frame pointers

The off-CPU stack trace collection, offcputime-bpfcc, requires the programs to be compiled with frame pointers for full backtraces.

OCaml

For OCaml, you’ll need a compiler variant with frame pointers enabled. If you are installing a released compiler using opam, you can create one the following switch command opam switch create 5.2.0+fp 5.2.0 ocaml-option-fp. Change out 5.2.0 for your preferred OCaml version.

Instead, if you are building the OCaml compiler from source, configure the compiler with --enable-frame-pointers option:

$ ./configure --enable-frame-pointers

Lastly, there is an option to create an opam switch with the development branch of the compiler. The instructions are in ocaml/HACKING.adoc. In order to create an opam switch from the current working directory, do:

$ opam switch create . 'ocaml-option-fp' --working-dir

glibc

The libc is not compiled with frame pointers by default. This will lead to many truncated stack traces. On Ubuntu, I did the following to get a glibc with frame pointers enabled:

1. Install glibc with frame pointers

   $ sudo apt install libc6-prof
   

2. LD_PRELOAD the glibc with frame pointers

   $ LD_PRELOAD=/lib/libc6-prof/x86_64-linux-gnu/libc.so.6 ./myapp.exe
   

Running

On one terminal run the program that you want to analyze:

$ LD_PRELOAD=/lib/libc6-prof/x86_64-linux-gnu/libc.so.6 ./ocamlfoo.exe

On another terminal run offcputime-bpfcc tool:

$ sudo offcputime-bpfcc --stack-storage-size 2097152 -p $(pgrep -f ocamlfoo.exe) 10 > offcputime.out

The command instruments the watches for 10s and the writes out the stack traces corresponding to blocking calls in offcputime.out. We use a large stack storage size argument so as to not lose stack traces. Otherwise, you will see many [Missing User Stack] errors in the back traces.

Caveats

offcputime-bpfcc must run longer than the program being instrumented by a few seconds so that the function symbols are resolved. Otherwise you may see [unknown] in the backtrace for function names.

Oddities

I still see an order of magnitude difference between the maximum pauses observed using offcputime-bpfcc and olly trace. Something is off.

Other links

• https://www.pingcap.com/blog/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance/

]]> KC Sivaramakrishnan https://nadh.in/blog/on-software-as-an-indiscipline/ https://nadh.in/blog/on-software-as-an-indiscipline/ Wed, 24 Jul 2024 00:00:00 GMT The nth-order effects of the recent CrowdStrike fiasco ^[1][2][3] will unfold over time. As it stands, it is apparently the single biggest global “tech outage” ever, which has already disrupted everything from airlines to railways to hospitals to financial systems amongst numerous others—globally.

]]> Kailash Nadh https://ibcomputing.com/it-admins-battle-windows-blue-screen-of-death-after-faulty-update/ https://ibcomputing.com/it-admins-battle-windows-blue-screen-of-death-after-faulty-update/ Fri, 19 Jul 2024 22:44:17 GMT

How IT Admins Are Fixing the Windows Blue Screen of Death

IT administrators worldwide are in crisis mode today due to a major issue with Windows computers. A recent faulty update from cybersecurity provider CrowdStrike has caused thousands of PCs and servers to crash with the notorious Windows Blue Screen of Death (BSOD) error. Although CrowdStrike has resolved the update issue, countless systems remain offline, affecting banks, airlines, supermarkets, and TV broadcasters.

The Fix: Tackling the Windows Blue Screen of Death

Windows Blue Screen of Death Fix  for many affected systems isn’t straightforward. IT admins are utilizing a suggested workaround from CrowdStrike, which involves booting Windows into Safe Mode and manually deleting the problematic system file to initiate the Windows Blue Screen of Death fix:

1. Boot Windows into Safe Mode or the Windows Recovery Environment.
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
3. Locate and delete the file matching “C-00000291*.sys”.
4. Reboot the host.

These steps help boot Windows into an environment where third-party drivers, such as CrowdStrike’s kernel-level driver, aren’t loaded. Admins must physically access machines to delete the faulty driver, complicating matters in environments with disk encryption like BitLocker or limited admin rights.

Challenges and Solutions

For many environments, the workaround is complicated by technical constraints, pushing some IT admins to wait for CrowdStrike’s fix. This entails repeated reboots in hopes that the update gets applied before the machine crashes again. Surprisingly, this “turn-it-off-and-on-again” method has had some success, with machines coming back online after multiple reboots. The delay can be attributed to CrowdStrike’s update servers being overwhelmed by an influx of requests from millions of machines.

Businesses using virtual desktops might recover faster by restoring systems to a pre-update state. However, in cases where reboots fail, booting into Safe Mode remains the most reliable option to implement a Windows Blue Screen of Death fix.

CrowdStrike’s Response

This crisis won’t be resolved in just a few hours. “It could take some time for systems that won’t automatically recover, but it is our mission to ensure every customer is fully operational,” stated CrowdStrike CEO George Kurtz in an interview with NBC News. Kurtz apologized for the widespread disruptions caused by the faulty update, promising a thorough investigation into how such an issue could affect so many machines globally.

The post IT Admins Battle Windows Blue Screen of Death After Faulty Update appeared first on IB Computing.

]]> Mujeeb cpy News https://ibcomputing.com/googles-goo-gl-link-shortener-expire-in-aug-2025/ https://ibcomputing.com/googles-goo-gl-link-shortener-expire-in-aug-2025/ Fri, 19 Jul 2024 22:30:54 GMT

Google’s Goo.gl Link Shortener Expire on Aug 2025

Hold onto your hyperlinks, folks!  Google’s Goo.gl Link Shortener Expire soon. it  is officially nearing the end of its digital road. After ceasing the creation of new goo.gl URLs in March 2019, Google has now set the definitive expiration date for these links: August 25, 2025.

The End of an Era

The sun began to set on goo.gl back in 2018 when Google first announced its shutdown for consumers and developers. A year later, the company halted the creation, analytics, and management of new links for all users. Fast forward to 2025, and it’s “404 Not Found” for any goo.gl URL still roaming the web.

Transition Period Ahead

Starting August 23, 2024, users clicking on a goo.gl link will encounter an interstitial page saying, “This link will no longer work in the near future.” Fear not, you can hit “Continue” to proceed, with an option to “Don’t show this again” for future clicks.

Developers, Take Note!

Google has a special note for developers: beware of potential disruptions! The interstitial page might throw a wrench in your redirect flows and social metadata displays. You might want to start shifting your goo.gl links ASAP. As an interim fix, adding the query parameter “si=1” should suppress the interstitial page for the time being.

Here’s the official word from Google:

“For example, if you are using other 302 redirects, the interstitial page may prevent the redirect flow from completing correctly. If you’ve embedded social metadata in your destination page, the interstitial page will likely cause these to no longer show up where the initial link is displayed. For this reason, we advise transitioning these links as soon as possible.”

The Next Step: Firebase Dynamic Links

Back in 2018, Google nudged developers to shift to Firebase Dynamic Links, designed to detect the user’s platform and direct them to either the web or an app. Oh, the irony! Google eventually pulled the plug on that service for developers too.

Time to Update Your URLs

So, mark your calendars for August 25, 2025 Google’s Goo.gl Link Shortener Expire. Whether you’re a developer or just a goo.gl enthusiast, it’s time to update those shortened links before they become digital relics.

The post Google’s Goo.gl Link Shortener Expire in Aug 2025 appeared first on IB Computing.

]]> Mujeeb cpy News https://ibcomputing.com/wikipedia-launches-long-awaited-dark-mode-feature/ https://ibcomputing.com/wikipedia-launches-long-awaited-dark-mode-feature/ Thu, 18 Jul 2024 21:43:01 GMT Dark Mode Finally Arrives

After years of anticipation, Wikipedia introduces dark mode, now available on select wikis for both mobile and desktop in reading and editing modes. This feature, one of the most requested by the community, aims to enhance accessibility and reduce eye strain by providing a low-contrast environment.

Why Dark Mode Matters

Dark mode hasn’t just been a nice-to-have; it’s been a necessity. Since at least 2019, editors and readers have pushed for this feature, with 20% signaling their dark mode preference at the OS level. Though research is mixed, many agree dark mode lessens eye strain and helps those with certain medical conditions.

Overcoming Challenges

Creating dark mode for Wikipedia involved overcoming significant technical hurdles. The new Vector 2022 skin was pivotal, and the collaborative effort between Wikimedia Foundation engineers and volunteer editors was essential. Articles like “International Orange” demonstrated the necessity of precision to avoid misinformation caused by altered color presentations.

The Accessibility Opportunity

Introducing dark mode highlighted pre-existing accessibility issues in many articles. For instance, the Wrexham A.F.C. article’s inaccessible red headers were revamped for better readability. This project has spotlighted the importance of wise color usage and overall content improvement.

How to Activate Dark Mode

Dark mode is currently available for logged-in users across all Wikipedias and gradually rolling out to non-logged-in users on wikis with minimal color issues or active technical editor involvement. Supported skins include Minerva (mobile) and Vector 2022 (desktop). Editors proficient in CSS and templates are encouraged to help expand dark mode availability.

A Collaborative Achievement

The rollout has been a collective effort, featuring invaluable contributions from volunteer editors and various Wikimedia teams. This milestone wouldn’t have been possible without their dedication and collaboration.

Enjoy the Experience!

As dark mode continues to expand, users are invited to enjoy a more comfortable reading experience on Wikipedia. For more info and to help improve accessibility, volunteer your expertise on CSS and templates.

Stay tuned as Wikipedia keeps evolving for the better – one dark mode at a time!

The post Wikipedia Launches Long-Awaited Dark Mode Feature appeared first on IB Computing.

]]> Mujeeb cpy News dark mode wikipedia https://ravidwivedi.in/posts/kenya-visa-process/ https://ravidwivedi.in/posts/kenya-visa-process/ Sun, 14 Jul 2024 10:54:02 GMT Prior to arrival in Kenya, you need to apply for an Electronic Travel Authorization (eTA) on their website by uploading all the required documents. This system is in place since Jan 2024 after the country abolished the visa system. The required documents will depend on the purpose of your visit, which in my case, was to attend a conference.

Here is the list of documents I submitted for my eTA:

• Scanned copy of my passport

• Photograph with white background

• Flight tickets (reservation)

• Hotel bookings (reservation)

• Invitation letter from the conference

• Yellow Fever vaccination certificate (optional)

• Job contract (optional)

“Reservation” means I didn’t book the flights and hotels, but rather reserved them. Additionally, “optional” means that those documents were not mandatory to submit, but I submitted them in the “Other Documents” section in order to support my application. After submitting the eTA, I had to make a payment of around 35 US Dollars (approximately 3000 Indian Rupees).

It took 40 hours for me to receive an email from Kenya stating that my eTA has been approved, along with an attached PDF, making this one of my smoothest experiences of obtaining travel documents to travel to a country :). An eTA is technically not a visa, but I put the word “visa” in the title due to familiarity with the term.

]]> Ravi Dwivedi https://www.evalapply.org/posts/poor-mans-job-runner-clojure-agents/index.html https://www.evalapply.org/posts/poor-mans-job-runner-clojure-agents/index.html Sun, 14 Jul 2024 00:00:00 GMT Aditya Athalye riff functional_programming clojure web_development https://ravidwivedi.in/posts/yellow-fever-vaccine/ https://ravidwivedi.in/posts/yellow-fever-vaccine/ Sat, 13 Jul 2024 07:56:03 GMT Recently, I got vaccinated with yellow fever vaccine as I am planning to travel to Kenya, a high risk country for yellow fever, in the near future. It should be taken at least 10 days before getting into the areas with yellow fever transmission to provide enough time for the formation of the antibodies. In order to get vaccinated, I searched for vaccination centers in Delhi and stumbled upon this page by the Indian government, which lists vaccination centers for yellow fever all over India. From that list, I made a phone call to the Airport Health Organization, a vaccination center near the Delhi Airport.

They asked me to write an email stating that I need yellow fever vaccination. After sending the email, they prompted me to attach a scanned copy of my passport data page, upon sending which they emailed me my appointment date, asking me to pay 300 INR in advance along with other instructions. The appointment date was 4 days after I sent scanned copy of my passport. The email also mentioned that those who are allergic to eggs or have never taken eggs should instead visit RML Hospital.

The vaccination center must be visited during 10 AM to 12 noon on the date of appointment. I reached there at around 11 AM and got vaccinated in around 40 minutes, followed by obtaining a vaccine certificate in half an hour.

One dose of this vaccine gives immunity against yellow fever for lifetime. Therefore, I can travel to any country with yellow fever transmission after getting this dose. Although some countries may require proof of vaccination within some time frame and some people might need a booster dose to maintain immunity.

]]> Ravi Dwivedi https://shrirangkahale.com/posts/thinking/ https://shrirangkahale.com/posts/thinking/ Mon, 08 Jul 2024 15:49:50 GMT Shrirang Kahale https://shrirangkahale.com/posts/living-offgrid/ https://shrirangkahale.com/posts/living-offgrid/ Fri, 14 Jun 2024 09:19:34 GMT Shrirang Kahale https://mrkaran.dev/posts/gullak/ https://mrkaran.dev/posts/gullak/ Fri, 14 Jun 2024 00:00:00 GMT A couple of weeks ago, I decided to start logging and tracking my expenses. The goal was not to record every minor purchase but to gain a general insight into where my money was going. In this post, I’ll dive deep into the behind-the-scenes of building Gullak—an expense tracker app with a dash of AI (yes).

Why#

My wife and I have a simple system for tracking our expenses during trips: we use Apple Notes to maintain a day-wise record, jotting down a one-liner for each expense under the date. This straightforward method has proven effective in keeping tabs on our spending habits while traveling.

For instance, during our last Europe trip, we recorded our daily expenses. After returning home, I was eager to analyze our spending patterns. I copied all these items into Google Sheets to analyse the top categories that I spent on during the trip.

I decided to develop a simple expense tracker app that automatically categorizes expenses into various groups like food, travel, shopping, etc. I believed this was a practical use case for leveraging an LLM paired with Function calling to parse and categorize expenses.

Initial Prototype#

The first step involved designing a prompt to capture user input about their spending. I picked up go-openai library and experimented with it.

Almost a year ago, I had developed a small bot for personal use, which provided a JSON output detailing the macronutrients and calories in specific food items, storing this information in Metabase. However, this was during the early days of API access provided by OpenAI. Due to occasionally unsatisfactory and inconsistent responses (despite instructions like “MUST RETURN JSON OR 1000 CATS WILL D*E SOMEWHERE”), it wasn’t entirely reliable.

Function calling addresses two main limitations of traditional language model responses:

• Inconsistent response format: Without function calling, responses from language models can be unstructured and inconsistent, requiring complex validation and parsing logic on the application side.
• Lack of external data integration: Language models are typically limited to the knowledge they were trained on, making it challenging to provide answers based on real-time or external data.

It’s important to note that the LLM does not actually execute any functions. Rather, we create a structure for the LLM to follow in its responses. The LLM would then generate a response with the content as a stringified JSON object following the schema provided in the function definiton.

I created a function called categorize_expense. This function takes a list of transactions as parameters, with each transaction having properties like transaction_date, amount, category, and description.

Here’s what this looks like:

fnCategorizeExpenses := openai.FunctionDefinition{
  Name:        "categorize_expense",
  Description: "Categorize expenses from the given input.",
  Parameters: jsonschema.Definition{
    Type: jsonschema.Object,
    Properties: map[string]jsonschema.Definition{
      "transactions": {
        Type:        jsonschema.Array,
        Description: "List of items purchased",
        Items: &jsonschema.Definition{
          Type: jsonschema.Object,
          Properties: map[string]jsonschema.Definition{
            "transaction_date": {
              Type:        jsonschema.String,
              Description: "Date of transaction in ISO 8601 format (e.g., 2021-09-01) if specified else today's date.",
            },
            "amount": {
              Type:        jsonschema.Number,
              Description: "Amount of the item",
            },
            "category": {
              Type:        jsonschema.String,
              Description: "One word category of the expense (e.g., food, travel, entertainment)",
            },
            "description": {
              Type:        jsonschema.String,
              Description: "Concise and short description of the item",
            },
          },
          Required: []string{"transaction_date", "amount", "category", "description"},
        },
      },
    },
    Required: []string{"transactions"},
  },
}

The response from this API call can then be unmarshalled into a struct.

var transactions models.Transactions

if err := json.Unmarshal([]byte(toolCall.Function.Arguments), &transactions); err != nil {
    return err
}

The next step was to determine exactly how users would provide input. I considered various methods that would make entering expenses as straightforward as my approach with Apple Notes and decided to create a Telegram bot.

I developed a Telegram bot that would parse the expenses and save them to a SQLite database. I explored tools like evidence.dev, a nice platform for creating frontends using the database as the sole source of truth. However, I encountered an issue where it could not correctly parse date values (see GitHub issue). Ultimately, I returned to my reliable old friend—Metabase.

However, I faced two main challenges with this approach:

• Privacy Concerns: Telegram does not offer the option to create a private bot; all bots generated through BotFather are public. To restrict access, I considered adding session tokens, but this approach was unsatisfactory. If I planned to distribute this bot, implementing a token-based, DIY authentication system on Telegram did not seem appropriate.

• Fixing Bad Entries: To correct erroneous entries, I had to manually update the SQLite table. As I intended to share this bot with my wife, I needed a more user-friendly workflow. Manually raw dogging UPDATE SQL queries was not the most user-friendly solution.

After a day or two of experimenting, I decided to build a small frontend for now.

Building Frontend#

As a backend developer, my core expertise is NOT JavaScript, and I strongly dislike the JS ecosystem. Obviously there’s no dearth of choices when it comes to frameworks, however for this project I wanted to stay away from the hype and choose a stack that is simple to use and productive (for me) out of the box. Having used Vue.js in production in the past, I feel it ticks those boxes for me as it comes bundled with a router, store, and all the niceties, and it has excellent documentation. After reading a refresher on the new Vue3 composition API syntax, I hit the ground running.

I find Tailwind CSS ideal for someone like me who prefers not to write CSS or invent class names. It’s a heavily debated topic online, but it’s important to pick our battles. An issue I encountered while researching UI frameworks was that Vue.js seems to have fewer options compared to React, likely due to its lower popularity. After some google-fu, I discovered a promising project called shadcn-vue, an unofficial community led port of the shadcn/ui React library.

The cool thing about this library is that it doesn’t come bundled as a package, meaning there’s no way to install it as a dependency. Instead, it gets added directly to your source code, encouraging you to tweak it the way you like.

I believe it’s an excellent starting point for anyone looking to build their own design system from scratch, as it allows for customization of both appearance and behavior. It might have been overkill for my simple UI, but I thought, what the heck, if side projects aren’t for exploring new things, what’s the point of it all? 😄

Database#

For the database, I opted for SQLite. It’s perfect for a small project like this since the database is just a single file, making it easier to manage. Initially, I used the popular driver mattn/go-sqlite3, but I found that the CGO-free alternative modernc/sqlite works just as well.

I also experimented with sqlc for the first time. For those unfamiliar, sqlc generates type-safe Go code from your raw SQL queries. It handles all the boilerplate database code needed to retrieve results, scan them into a model, manage transactions, and more. sqlc makes it seem like you’re getting the best of both worlds (ORM + raw SQL).

Here’s an example query:

-- name: CreateTransaction :many
-- Inserts a new transaction into the database.
INSERT INTO transactions (created_at, transaction_date, amount, currency, category, description, confirm)
VALUES (?, ?, ?, ?, ?, ?, ?)
RETURNING *;

Using sqlc generate, it generates the following code:

// Code generated by sqlc. DO NOT EDIT.
// versions:
//   sqlc v1.26.0
// source: queries.sql

package db

import (
	"context"
	"database/sql"
	"time"
)

const createTransaction = `-- name: CreateTransaction :many
INSERT INTO transactions (created_at, transaction_date, amount, currency, category, description, confirm)
VALUES (?, ?, ?, ?, ?, ?, ?)
RETURNING id, created_at, transaction_date, currency, amount, category, description, confirm
`

type CreateTransactionParams struct {
	CreatedAt       time.Time `json:"created_at"`
	TransactionDate time.Time `json:"transaction_date"`
	Amount          float64   `json:"amount"`
	Currency        string    `json:"currency"`
	Category        string    `json:"category"`
	Description     string    `json:"description"`
	Confirm         bool      `json:"confirm"`
}

// Inserts a new transaction into the database.
func (q *Queries) CreateTransaction(ctx context.Context, arg CreateTransactionParams) ([]Transaction, error) {
	rows, err := q.query(ctx, q.createTransactionStmt, createTransaction,
		arg.CreatedAt,
		arg.TransactionDate,
		arg.Amount,
		arg.Currency,
		arg.Category,
		arg.Description,
		arg.Confirm,
	)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	items := []Transaction{}
	for rows.Next() {
		var i Transaction
		if err := rows.Scan(
			&i.ID,
			&i.CreatedAt,
			&i.TransactionDate,
			&i.Currency,
			&i.Amount,
			&i.Category,
			&i.Description,
			&i.Confirm,
		); err != nil {
			return nil, err
		}
		items = append(items, i)
	}
	if err := rows.Close(); err != nil {
		return nil, err
	}
	if err := rows.Err(); err != nil {
		return nil, err
	}
	return items, nil
}

Apple Shortcuts#

Similar to my Apple Notes approach, I wanted to create a shortcut that would allow me to log expenses quickly. I created a simple shortcut that would prompt me to enter the expenses and send an HTTP POST request to Gullak’s API server. I then open the dashboard once in a while to confirm/edit these unconfirmed transactions.

You can read more about setting up the Shortcut in your Apple devices here.

Proudly, Not a Weekend Project#

For every “I could do this in a weekend” comment, yes, this project is straightforward—a “CRUD GPT” wrapper that isn’t complicated to build. Yet, it took me over a month to develop. I spent less than an hour most days on this project, instead of cramming it into an all-nighter weekend project - an approach I want to move away from. Slow and steady efforts compound, outlasting quick, sporadic bursts. I’m pleased to balance this with my full-time job without burning out.

Ideas for the Future#

Initially, I didn’t set out to build a comprehensive budgeting app, just an expense logger, as that was my primary need. However, if usage increases and the tool proves helpful in reducing unnecessary spending, I’m open to adding more features. Some possibilities include a subscription tracker, integration with budgeting tools like YNAB or Actual through their APIs, and monthly reports sent via email. The best part is that you own complete data, as the data is stored locally on your device so you can also export it anytime and build other integrations on top of it.

Feel free to open a GitHub issue or reach out if you have any suggestions or feedback. I’m excited to see where this project goes!

Update (Feb 2026)#

I’ve rewritten Gullak to use plain-text ledger files. Read more about why in the follow-up post: Why Plain-Text Ledger is Powerful for Gullak.

]]> Karan Sharma https://www.learningfromdata.zingg.ai/p/hello-zingg-id https://www.learningfromdata.zingg.ai/p/hello-zingg-id Sat, 08 Jun 2024 11:50:27 GMT

Imagine we have multiple records of a customer spread across ticketing, billing, offline stores, and web properties. This is a usual scenario we witness everyday in our data land. Different sources of data come with multiple versions of truth. When they land centrally in a warehouse or datalake, its hard to figure things out. Revenue attribution and lifetime value can not be calculated accurately. Nor can we estimate churn or calculate confidently how many new customers were added last quarter. Increasing revenue through personalized offers and exploring opportunities to cross sell and upsell will not be possible either. Let alone anti money laundering or risk. Visualize a multimillion-dollar investment in a GenAI customer service agent. How far can it go using this fragmented data?

To establish the right data foundation, analyze and report on business critical metrics, and productionize AI and ML technology, we will need to unify such records and build a trusted single source of truth for the customer.

Enter entity resolution. Variations in attributes can all be resolved, and clusters of matching records can be created. The Zingg Community Version offers robust clustering and fuzzy matching, which bring these records together with a unique Z_CLUSTER field. Matching records share the same Z_CLUSTER. Voila, there is a way to say who is who!

What happens when we want to track the customer over time? Or build a holistic view? Establish their journey and reference them or search for them in different enterprise systems. When there is no identifier to refer to the customer, we need a way to associate them with an identifier. Kind of like an SSN or a passport number. Or a primary key in the database.

The Z_CLUSTER is extremely powerful and useful. But it is not persistent. There is no guarantee that it will not change in the next Zingg run. If we stream our resolved records into our source systems and want to use Z_CLUSTER as the primary key, it would break.

As we saw different Zingg deployments, this was an ask that we heard again and again. A globally unique persistent identifier that acts like a reference key to the entity and all its records. The same key even when the record changes. The unambigious way to refer to an entity across multiple enterprise data systems, operational processes, and workflows. Even as new records come in and earlier ones get updated.

This is exactly the ZINGG_ID in the Zingg Enterprise version. Resolved, unified records across the entire enterprise which can be referenced through a persistent key. A help desk agent could use it to look up the entire customer interaction journey. A GPT based chatbot operating on patient data would work seamlessly with all the information at its disposal.

It's been fun solving this complex piece of the puzzle, and it's extremely satisfying to witness the value it generates for Zingg Enterprise customers.

Sounds interesting, or something you care about? Do get in touch!

]]> Sonal Goyal https://saket-choudhary.me/blog/2024/04/28/augustbirthdays/ https://saket-choudhary.me/blog/2024/04/28/augustbirthdays/ Sat, 27 Apr 2024 18:30:00 GMT Autumn > Summer > Monsoon. I am sure you are thinking about heterogeneity, what does this relationship look like if we focus on each state individually? When I broke down the association analysis for each state individually and arranged the states based on the strength of the correlation (between the relative percentage of conceptions every month and the mean temperature in each state), a beautiful pattern emerged. 24 out of 28 states for which I had data show a correlation coefficient of -0.5 or lower (that is, the absolute strength of correlation exceeds 0.5). For states like Manipur, Bihar, and Haryana the association between temperature and rate of conception is as high as 0.91, implying a higher temperature is associated with fewer conceptions and a one-degree drop in temperature will lead to an increase of 0.91% in conception assuming this relationship is indeed causal. States like Jammu and Kashmir and Uttarakhand which are usually colder have weak associations. While Kerala, which has a tropical climate, has a stronger association with a correlation coefficient of -0.82. Thus, the association is not as strong for colder climates, an observation that I previously made at the season level in my previous plot. Causality is hard to prove here. We have strong associations that reproduce across states, Occam’s razor, and the golden fact that correlation does not imply causation. I wish the answer was as simple as those birthday parties. Summer has come and passed The innocent can never last Wake me up when September ends - Billie Armstrong]]> One of my fond memories while growing up in Rawatbhata, a small town (and now a city!) of Rajasthan was birthday parties. These birthday parties used to be simple. We were all a bunch of similar-aged kids and would just invite each other for a mini celebration.

Everything about these parties was simple. The people were simple. The invitation was simple - you visited each house and verbally invited everyone. The food was simple. There were no starters, a couple of large bottles of ThumpsUp would suffice and was often served in steel glasses which would come out just for this occasion. There was only one simple course - the main course with pulao (rice + sautéed veggies), poori (fried flatbread), dessert (gulab jamun or kheer), chole (chickpeas), and well, a piece of cake. Paneer had not entered the menu yet, though it was sometimes on the menu if you were invited home for dinner.

The music was simple and the dance was simple. Bollywood’s top four dance numbers would play on a loop from a Philips cassette player that everyone had. It was available at a discount for everyone or I recall. And yes, there were simple “return gifts” - a Nataraj pencil and a Non-dust eraser. Simple times.

There were no bakeries in Rawatbhata for a long time, and almost all cakes were home-baked. No one in the town had ovens! There were no microwaves either. The only electrical gadget in the kitchen used to be a mixer grinder. How do you “bake” without an “oven”? The technique uses a simple idea - use sand to provide controlled heating. Too much digression. But the cakes were simple too.

Baking the cake on sand and stove! Source

While we would get invited to all the birthdays, I do remember sneaking in uninvited to a few, once in a while. I also distinctly remember August being a month of celebrations. Of course, there was Independence Day on the 15th lined up with five birthdays in August! Five! Most other months would have one or two. Naturally, August was also my favourite month - loads of good food and return gifts. People with birthdays in August had to be extra diligent. While the simple birthday menu rarely changed, the “return gifts” required deeper thought. You did not want to be called out for repeating a return gift in the same month.

Caught in the nostalgia of food, birthdays, and 90s music, I had a question going around in my head: Were the five birthdays in August a rare event? There are 365 days in a year and assuming no day is special, there is no reason August should be a non-simple birthday month. How do I find out if there is anything special about August, if there is? My first thought was to poll my odd group of college friends and ask them for their birthdays. However hard I try to avoid it, this would be a convenient sample, and I don’t think I would have learned anything mechanistic about the “why” or “how”.

What do I need to answer my question? My question can be asked more simply: when do most Indians celebrate their birthdays? If I had access to all the Aadhaar data, this question could be answered using probably a few lines of code. Aadhaar is of course closed for these use cases. A little bit of search landed me on a wonderful resource of HMIS. The description on the website is self-explanatory:

This portal will be a gateway to wealth of information regarding the health indicators of India. The information available on this portal is derived data from data uploaded by the States/ UTs. HMIS data is specifically designed to support planning, management, and decision making based on on Grading of facilities, various indicators at Block, District at State as well as National Level.

While the emphasis is on “health indicators”, HMIS has district and state-level data on how many births happen in private and public hospitals. Getting this data was quite an exercise and taught me several tricks for parsing excel/html. After struggling disproportionately with weirdly formatted files, I could extract all the birth data between 2008 and 2020 across states. With this data in hand, I could finally answer the simple question: was august the special birthday month?

I first looked at the distribution of births. September, October, and August have the highest number of births with approximately 1.9 million average births over 2008-2020. Since births peak in these months, there is an inherent “seasonality” attached to birthdays in India. In a hypothetical world, all the months would have roughly equal births, rather than having a range from 1.38 million births (April) to 1.98 million births (September).

Taking an average can sometimes hide a tonne of information that lies in a time-series data. If we look at the entire period between January 2008 and December 2020, the “seasonality” is easier to spot. From 2008 to 2020, the birth curve rises and dips over the year. The peaks happen around September/October, while April registers a deep dip.

While the data from HMIS is on births, we can infer the time of conception by simple arithmetic - subtracting 9 months. Of course, the seasonality remains intact. The peak of conceptions happens in December of the previous year or January of the same year.

Summarizing information at the country level is a good starting point. Overall, September is the month with the highest number of birthdays. But does that hold across all states? What about Rajasthan? I next broke it down at the state level and annotated the month with the highest number of birthdays.

From the figure above, August is the month with the highest number of birthdays in Rajasthan. But more importantly, the figure highlights the diversity that underlies India. While September, the month with the highest number of total births, is the month of peak births in 9 states, October is the peak month in 10 states. But there are also Meghalaya and Tripura which peak in January and December respectively. On the other extreme, July is the month of the least conceptions across multiple states. We can flip the births and look at the conception curve.

While my actual question was answered, and I discovered that there is heterogeneity within the country in how it celebrates birthdays, my related question of understanding why this happens “mechanistically” remained unanswered. I must admit beforehand that it is also a hard question to answer without access to a tonne of data.

Why are conceptions higher in a month? Why do they vary across states? Is it driven by the “wedding season” in the country?

I did not have access to the wedding registration data. So I asked a simple question: does temperature affect the rate of conception (and hence birth) in India. Surprisingly, getting temperature data for a city across a time span without paying anyone remains a non-trivial task. However after a bit of tussle and jumping language hoops, I was able to download the gridded temperature data from IMD, Pune.

To understand the relationship between temperature and rate of conceptions, I looked at the average temperature over 2008-2020 in a state and calculated its correlation with the number of estimated conceptions. For the autumn months (October/November), the correlation coefficient is the highest (-0.56). But even more importantly, this figure hinted towards the seasonality being associated with temperature. The rate of conceptions across seasons varies as Winter > Autumn > Summer > Monsoon.

I am sure you are thinking about heterogeneity, what does this relationship look like if we focus on each state individually?

When I broke down the association analysis for each state individually and arranged the states based on the strength of the correlation (between the relative percentage of conceptions every month and the mean temperature in each state), a beautiful pattern emerged. 24 out of 28 states for which I had data show a correlation coefficient of -0.5 or lower (that is, the absolute strength of correlation exceeds 0.5). For states like Manipur, Bihar, and Haryana the association between temperature and rate of conception is as high as 0.91, implying a higher temperature is associated with fewer conceptions and a one-degree drop in temperature will lead to an increase of 0.91% in conception assuming this relationship is indeed causal. States like Jammu and Kashmir and Uttarakhand which are usually colder have weak associations. While Kerala, which has a tropical climate, has a stronger association with a correlation coefficient of -0.82. Thus, the association is not as strong for colder climates, an observation that I previously made at the season level in my previous plot.

Causality is hard to prove here. We have strong associations that reproduce across states, Occam’s razor, and the golden fact that correlation does not imply causation. I wish the answer was as simple as those birthday parties.

Summer has come and passed  
The innocent can never last  
Wake me up when September ends

                - Billie Armstrong

---

]]> Saket Choudhary https://mrkaran.dev/posts/random-kind-stranger/ https://mrkaran.dev/posts/random-kind-stranger/ Sun, 21 Apr 2024 00:00:00 GMT Last month, I did a wonderful trip travelling through the scenic landscapes of Switzerland. My wife and I were in Lucerne and had scheduled a day trip to Mt. Titlis for the next day but were wondering what to do that evening. After strolling along the Chapel Bridge and enjoying an amazing lunch by the waterfront, my wife and I collectively decided to book a Lake Lucerne cruise for the evening. It seemed like the perfect setup for a romantic date night, or so I thought!

We got the tickets from the information booth for the evening and went back to our hotel to freshen up and relax for a bit.

We arrived exactly at 6:45 PM, as mentioned on our tickets, and started to wait. Except, there were only the two of us waiting. We waited for probably half an hour, until 7:15 PM, which was the departure time. Knowing how precise the Swiss transport system usually is, I sensed something fishy. Luckily, I spotted Martin, a staff member of the cruise company, and asked him about it. He looked puzzled by my question and informed me that there was no cruise scheduled for today. Yep, not today, not tomorrow, and not for the rest of the weekend. It was Easter time, and all the cruise trips were cancelled. In fact, he was quite as puzzled as I was as to how the lady at the ticket counter even gave us the tickets for today. However, he told me that he couldn’t do much and suggested writing an email for a refund. I was a bit sad as this dashed our evening plans, but I thought, fine… Shit happens. It’s not the end of the world!

And then, out of nowhere, by pure serendipity, my wife spotted another member of the cruise—this time, in fact, the captain! She began to tell the captain about our ordeal. The captain, a very warm and kind lady, listened to my wife patiently and understood our plight! She apologized on behalf of her company and immediately offered a refund in cash for the tickets that we had purchased. We were quite happy, as following up on emails, etc., was something that I was not particularly excited about looking forward to during the other half of my trip. So we said yes, except she needed to go to the ATM to draw cash. We waited for her and chatted with Martin about random stuff! He told us some really fun stories about how Toblerone’s iconic packaging no longer features the Matterhorn mountain. He also shared practical advice on how to safeguard oneself against pickpockets in Italy(we were gonna visit it soon), and reminisced about his life in Zermatt before relocating to Lucerne.

Anyway, the wait stretched longer than usual, and I found myself wondering about her whereabouts. And then finally, we saw her approaching us. She told us that she hadn’t found a working ATM nearby, and had to go a bit far. But she didn’t come back empty-handed; she also brought us macaroons as a token of her apologies, a gesture that was incredibly thoughtful. She handed us 200 CHF in cash, covering more than the 180 CHF we had paid for our tickets. When we attempted to return the excess 20 CHF, she firmly refused to take it back. Despite our insistence on returning the excess 20 CHF, she remained steadfast, refusing to accept it. She encouraged us to use the extra money to enjoy a few drinks, suggesting it as a consolation for our evening plans being spoiled by the canceled cruise.

I am glad the cruise didn’t happen. Life has its own ways of revealing that kindness exists in every corner of the world, and serendipity can lead to the most memorable encounters! For most, it may not be a huge thing, but for me, it was a profoundly touching and generous act from a stranger who simply chose to be kind without any ulterior motives.

The lesson: Be kind to others, do no harm, and always pay it forward. :)

Fin!

]]> Karan Sharma https://shrirangkahale.com/posts/dying-excitement-of-festivals/ https://shrirangkahale.com/posts/dying-excitement-of-festivals/ Sat, 06 Apr 2024 13:09:55 GMT Shrirang Kahale https://www.evalapply.org/posts/writing-maketh-the-10x-developer/index.html https://www.evalapply.org/posts/writing-maketh-the-10x-developer/index.html Fri, 05 Apr 2024 00:00:00 GMT Aditya Athalye riff meta whyto writing culture tools_for_thought programming https://ravidwivedi.in/posts/taj-mahal/ https://ravidwivedi.in/posts/taj-mahal/ Fri, 29 Mar 2024 10:13:21 GMT Introduction

I visited the Taj Mahal this month with my friend Badri. Taj Mahal is a major tourist attraction, getting 7-8 million visitors per year. It is one of the most beautiful pieces of architecture.

Journey to Agra

Taj Mahal is in the city of Agra, which is 188 km from Delhi by train. We had booked a train by the name of Taj Express from Hazrat Nizamuddin station in Delhi to Agra Cantt railway station. The night before the journey, we stayed in a retiring room at Old Delhi railway station because we could not get one at Hazrat Nizamuddin station. Retiring rooms are accommodations for the passengers at the railway stations, which require a confirm train ticket to stay.

Of all the retiring rooms I have been to, locating the ones at Old Delhi station was the most challenging. I recommend reading Badri’s blog post, where he details our experience with this and other retiring rooms.

The next day, we barely reached Hazrat Nizamuddin station in time for the train. However, the train had not yet arrived at the station and was delayed by half an hour.

Arrival in Agra

We reached Agra Cantt station at 10:30 hours, where we stayed in a retiring room. Before heading out to Taj Mahal, we rested in our rooms for a couple of hours, followed by having lunch at the station. As we came out of the station, we were approached by an autorickshaw driver who quoted the price to Taj Mahal as 150 INR. I negotiated the price down to 60 INR, which the driver agreed to on the condition of sharing the ride with other passengers. When we realized he was not making any effort to bring in more passengers to share the ride with, we walked away.

As we came out of the station complex, an autorickshaw driver offered a ride to the Taj Mahal for 20 INR per person on a sharing basis, or 100 INR for a reserved ride. We agreed to share the ride with other passengers for 20 INR, but the driver started driving as soon as we got in. We mistook the other person in the autorickshaw for a passenger we were sharing the ride with, but later found out that they were with the driver. Upon reaching the outer gate of the Taj Mahal, we paid him 40 INR as agreed, but he insisted that we had reserved the auto. We told him that we did not reserve the auto, but rather chose the sharing option. He insisted on this for some time. I suspect it was a scam, but we just walked away, and he didn’t pursue us further.

Exploring the Taj Mahal

The autorickshaw dropped us at one of the outer gates, from where we had to walk about 500 meters to reach the ticket counter just outside the west gate. We bought tickets worth 250 INR per person, which also allowed us to enter the mausoleum. Then we proceeded to the security check before entering the Taj Mahal complex.

Upon entering, we saw red sandstone walls on three sides enclosing the Taj Mahal complex. We took photos using my phone and a Fujifilm camera. Later, we learned that we needed to cover our shoes before entering the mausoleum. We also saw a few people barefoot, but we couldn’t find a place to leave our shoes. We came out of the whole complex at 18:00 hours and had snacks with tea at a nearby shop. I also bought a fridge magnet as a souvenier for 30 INR.

Our next destination was Jaipur and we had booked our seats in a train from Agra Cantt station. We decided to walk towards the station, hoping to have dinner along the way. However, we could not find a place to eat. Since we enjoyed the lunch at the station earlier, we went there for dinner as well, after which we boarded our train to Jaipur. On our way back to the station, I found the bus station for the Taj Mahal.

Expenses

These were our expenses per person:

Description	Amount (Indian Rupees)
Retiring room at Delhi Railway Station (12 hours)	131
Train ticket from Delhi to Agra (Taj Express)	110
Retiring room at Agra Cantt station for 12 hours:	450
Auto-rickshaw to Taj Mahal	20
Taj Mahal ticket (including entry to the mausoleum)	250
Food	350
Total	1,311

]]> Ravi Dwivedi https://mrkaran.dev/posts/travel-tailscale/ https://mrkaran.dev/posts/travel-tailscale/ Wed, 27 Mar 2024 00:00:00 GMT I have an upcoming trip to Europe, which I am quite excited about. I wanted to set up a Tailscale exit node to ensure that critical apps I depend on, such as banking portals continue working from outside the country. Tailscale provides a feature called “Exit nodes”. These nodes can be setup to route all traffic (0.0.0.0/0, ::/0) through them.

I deployed a tiny DigitalOcean droplet in BLR region and setup Tailscale as an exit node. The steps are quite simple and can be found here.

$ echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
$ echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
$ sudo sysctl -p /etc/sysctl.d/99-tailscale.conf
$ sudo tailscale up --advertise-exit-node

The node is now advertised as an exit node, and we can confirm that from the output of tailscale status:

$ sudo tailscale status                       
100.78.212.33   pop-os               mr-karan@    linux   -
100.75.180.88   homelab              mr-karan@    linux   -
100.100.191.57  iphone               mr-karan@    iOS     offline
100.123.189.14  karans-macbook-pro   mr-karan@    macOS   offline
100.104.67.7    lab                  mr-karan@    linux   offline
100.108.220.87  tailscale-exit       mr-karan@    linux   active; exit node; direct 167.71.236.222:41641, tx 21540 rx 17356

On the client side, I was able to start Tailscale and configure it to send all the traffic to the exit node with:

sudo tailscale up --exit-node=100.108.220.87

We can confirm that the traffic is going via the exit node by checking our public IP from this device:

➜ curl  https://ipinfo.io 
{
  "ip": "167.x.x.222",
  "city": "Doddaballapura",
  "region": "Karnataka",
  "country": "IN",
  "loc": "13.2257,77.5750",
  "org": "AS14061 DigitalOcean, LLC",
  "postal": "560100",
  "timezone": "Asia/Kolkata",
  "readme": "https://ipinfo.io/missingauth"
}                            

However, I encountered a minor issue since I needed to bring my work laptop for on-call duties, in case any critical production incidents required my attention during my travels. At my organization, we use Netbird as our VPN, which, like Tailscale, creates a P2P overlay network between different devices.

The problem was that all 0.0.0.0 traffic was routed to the exit node, meaning the internal traffic meant for Netbird to access internal sites on our private AWS VPC network was no longer routed via the Netbird interface.

Netbird automatically propagates a bunch of IP routing rules when connected to the system. These routes are to our internal AWS VPC infrastructure. For example:

10.0.0.0/16 via 100.107.12.215 dev wt0

Here, wt0 is the Netbird interface. So, for example, any IP like 10.0.1.100 will go via this interface. To verify this:

$ ip route get 10.0.1.100
10.0.1.100 dev wt0 src 100.107.12.215 uid 1000 

However, after connecting to the Tailscale exit node, this was no longer the case. Now, even the private IP meant to be routed via Netbird was being routed through Tailscale:

$ ip route get 10.0.1.100
10.0.1.100 dev tailscale0 table 52 src 100.78.212.33 uid 1000 

Although Tailscale nodes allow for the selective whitelisting of CIDRs to route only the designated network packets through them, my scenario was different. I needed to selectively bypass certain CIDRs and route all other traffic through the exit nodes. I came across a relevant GitHub issue, but unfortunately, it was closed due to limited demand.

This led me to dig deeper into understanding how Tailscale propagates IP routes, to see if there was a way for me to add custom routes with a higher priority.

Initially, I examined the IP routes for Tailscale. Typically, one can view the route table list using ip route, which displays the routes in the default and main tables. However, Tailscale uses routing table 52 for its routes, instead of the default or main table.

$ ip route show table 52                                                           

default dev tailscale0 
100.75.180.88 dev tailscale0 
... others ...
throw 127.0.0.0/8 
192.168.29.0/24 dev tailscale0 

A few notes on the route table:

• default dev tailscale0 is the default route for this table. Traffic that doesn’t match any other route in this table will be sent through the tailscale0 interface. This ensures that any traffic not destined for a more specific route will go through the Tailscale network.

• throw 127.0.0.0/8: This is a special route that tells the system to “throw” away traffic destined for 127.0.0.0/8 (local host addresses) if it arrives at this table, effectively discarding it before it reaches the local routing table.

We can see the priority of these IP rules are evaluated using ip rule show:

➜ ip rule show          
0:	from all lookup local
5210:	from all fwmark 0x80000/0xff0000 lookup main
5230:	from all fwmark 0x80000/0xff0000 lookup default
5250:	from all fwmark 0x80000/0xff0000 unreachable
5270:	from all lookup 52
32766:	from all lookup main
32767:	from all lookup default

This command lists all the current policy routing rules, including their priority (look for the pref or priority value). Each rule is associated with a priority, with lower numbers having higher priority.

By default, Linux uses three main routing tables:

• Local (priority 0)
• Main (priority 32766)
• Default (priority 32767)

Since Netbird already propagates the IP routes in the main routing table, we only need to add a higher priority rule to lookup in the main table before Tailscale takes over.

$ sudo ip rule add to 10.0.0.0/16 pref 5000 lookup main

Now, our ip rule looks like:

$ ip rule show          
0:	from all lookup local
5000:	from all to 10.0.0.0/16 lookup main
5210:	from all fwmark 0x80000/0xff0000 lookup main
5230:	from all fwmark 0x80000/0xff0000 lookup default
5250:	from all fwmark 0x80000/0xff0000 unreachable
5270:	from all lookup 52
32766:	from all lookup main
32767:	from all lookup default

To confirm whether the packets for destination 10.0.0.0/16 get routed via wt0 instead of tailscale0, we can use the good ol’ ip route get:

$ ip route get 10.0.1.100 
10.0.1.100 dev wt0 src 100.107.12.215 uid 1000

Perfect! This setup allows us to route all our public traffic via exit node and only the internal traffic meant for internal AWS VPCs get routed via Netbird VPN.

Since, these rules are ephemeral and I wanted to add a bunch of similar network routes, I created a small shell script to automate the process of adding/deleting rules:

#!/bin/bash

# Function to add IP rules for specified CIDRs
add() {
    echo "Adding IP rules..."
    sudo ip rule add to 10.0.0.0/16 pref 5000 lookup main
    # ... others ...
}

# Function to remove IP rules based on preference numbers
remove() {
    echo "Removing IP rules..."
    sudo ip rule del pref 5000
    # ... others ....
}

# Check the first argument to determine which function to call
case $1 in
    add)
        add
        ;;
    remove)
        remove
        ;;
    *)
        echo "Invalid argument: $1"
        echo "Usage: $0 add|remove"
        exit 1
        ;;
esac

Fin!

]]> Karan Sharma https://ravidwivedi.in/posts/thailand-trip/ https://ravidwivedi.in/posts/thailand-trip/ Thu, 21 Mar 2024 20:45:00 GMT Arrival

I was staying at a hostel in Bukit Bintang, a shopping hub and entertainment center of Kuala Lumpur, which was like 200 metres from the monorail station, where I took a metro train for KL Sentral. Then I took a bus from KL Sentral to the airport, followed by taking a Malaysian Airlines flight to Bangkok. Meals were included without any extra charge.

The flight took around 2 hours, landing in Bangkok’s Suvarnabhumi Airport at around 15:00 hours local time. My friend Fletcher was leaving for Pattaya from Bangkok, which is about 150 km and 2 hours bus ride, but takes a longer time from within the Bangkok city compared to Suvarnabhumi Airport due to the traffic in Bangkok. We had planned to meet in Pattaya and had already booked a hostel there. At the bus counter, I came to know that buses leave for Pattaya every hour. However, the next bus for which tickets were available was at scheduled to leave at 19:00 hours.

The ticket cost was 143 Thai Baht (360 Indian Rupees) per seat. This left me with a couple of hours at the airport, which I spent charging my phone, exploring shops and eating snacks which I had packed. As I had one week to spend in Thailand, buying a SIM card was essential to have internet access on the go. The internet plans I checked at the airport were expensive compared to Kuala Lumpur, so I didn’t buy a SIM card from there, leaving me with no way to coordinate with Fletcher.

Pattaya

My bus left airport at 7 PM and dropped me at the last stop named Jomtien beach in Pattaya at around 9 PM. The bus journey was smooth, thanks to good roads and lack of traffic jam. I used OsmAnd app for navigation and decided to walk to my hostel as it was around 1 km away. While walking, I noticed Cannabis bars on the street side and ubiquitous massage parlors, not to mention the street prostitutes vying for your attention.

After some time, I realized that the address mentioned on the hostel receipt was different from the one OpenStreetMap was showing, leading me to ask a person sitting at a café for help. That person agreed to help me and took me to the exact place where the hostel was. On the way, he told me he was originally from Kuwait but lived in Pattaya for many years. He also told me about the shared hail-and-ride cheap songthaew rides which ran along the Jomtien Beach, charging 10 Baht (25 Rupees) for any distance along that route, making our trip cost-effective.

I was at my hostel, but I was still not sure if I was at the correct place. There was no reception in sight, but only a staircase. In the meanwhile, I bumped into Fletcher, which gave me a sigh of relief. Adjacent to the property was a hairdresser who helped us get into the hostel. This was my first time checking in to a hostel without a reception. It seemed like a shared room without service. After some time, we went outside and bought a SIM card for me at a 7-Eleven store. The cost was 399 Baht for 7 day unlimited internet.

Next day, we went to Pattaya Tiger Park where you can go inside the tiger’s cage and touch the tiger. Then we went to the nearby Floating Market but decided against going inside after reviewing ticket prices which were higher than we deemed worthwhile. After this, we walked towards the Jomtien beach, but I planned to hitchhike as it was very sunny, and our drinking water supply was getting depleted. A few cars passed by without responding, but a scooty stopped and picked us up. Initially, I thought he would let only one of us ride, but to my surprise he let both of us on it. During the ride, he told us he is from Delhi and dropped us at walking distance from the bus station where Fletcher had to board his bus for Bangkok.

In the evening, I went to a shop which was selling fruits like guavas, dragon fruits, mangoes. I sampled guavas, which I found tasty and got mangoes and dragon fruits packed for eating later, and they were yummy.

Bangkok

Next day, on 9th Feb, I left for Bangkok, taking the bus at the bus station where the bus from Suvarnabhumi Airport dropped me the other day. I reached the airport in 2 hours and boarded a metro train from there to Huai Khwang, followed by walking to my hotel. This hotel room costed 5600 INR for four days for a double bed. In Bangkok too, I found abundance of convenience stores like 7-Eleven, similar to Pattaya and Kuala Lumpur. I had difficulty finding vegetarian food and labels were in Thai language, and the staff didn’t know English.

Next day, on 10th Feb 2024 was Chinese New year. I didn’t see a lot of celebrations in my area, but I saw celebratory programs and discount sales while I was in Kuala Lumpur, a few days before the Chinese New Year. In terms of food, I mainly relied on fruits. The pineapples were different from what I have seen in India, and this was the first time I saw banana flesh yellow. I tried Korean vegetable noodles, which were good. Another option was eating bread and cheese from the 7-Eleven. I also tried some Indian food like Chhole Kulche at Sukhumvit and some paneer dish at Ratchada Night market.

I explored malls in Siam and explored the area around Sukhumvit. The highlight of my Bangkok trip was the Chao Phraya Express Boat, which sails on the Chao Phraya River and covers some important tourist destinations of Bangkok. I took the all day river pass for the boat for 150 Thai Baht, which allows you to hop on and off any boat at any station for the day, only to realize later that it would have been better to buy respective station tickets to reduce the costs. I highly recommend taking the boat ride as it offers stunning views of the city. I went to Wat Arun, a famous Buddhist temple and a major tourist attraction. Upon entering the temple after buying the ticket, they stamped my hand ;) Following this, I took a boat to the famous Khao San Road, which is famous for cheap accommodations and food stalls.

I had booked a flight from Bangkok to Delhi with Air India, and during the flight, they were serving alcohol onboard. I decided to try red wine, marking my first experience of consuming alcohol while flying.

Notes

• There are many malls in Bangkok and you can easily find toilets/restrooms, which can help to avoid paying for using washrooms.

• Compared to Malaysia, I found Thailand more expensive, contrary to expectations. In addition, shopping in Malaysia was quite cheap due to Chinese New Year discounts, which I didn’t see in Thailand.

• I liked Pattaya more than Bangkok, mainly due to Pattaya having a nice beach and I could engage with locals and foreigners. Perhaps booking a hostel instead of a hotel in Bangkok would have given more opportunities to engage with people.

]]> Ravi Dwivedi https://www.learningfromdata.zingg.ai/p/deterministic-matching-vs-probabilistic https://www.learningfromdata.zingg.ai/p/deterministic-matching-vs-probabilistic Sat, 16 Mar 2024 14:09:14 GMT

In the entity resolution space, we refer to exact matching on attributes as deterministic matching. If the application systems are consistently collecting and persisting trusted and known identifiers like Social Security Number, deterministic matching can help to resolve such records and help us identify the true individual.

In most real world systems, data in different systems in an organisation has different attributes. Let us imagine a retail store with both online and offline presence. A web support system would capture the email correctly while also logging name, a call centre would mostly have a correct phone number but other details may not be as reliable. The offline store would have some or all of the details, as accurately as the human behind the computer tasked with entering them can type it in. Thus some of the individual records have identifiers, like phone and email in the above case, which may match exactly, while many of the records have names and addresses with all kinds of variations, typos, abbreviations etc. Assessing the extent of such variations for an entity and reconciling the fuzzy and inexact matches is known as Probabilistic Matching.

At a simplistic level, one can think of deterministic matching as a database join on trusted id columns like passport number or email. Probabilistic Matching involves statistical and ML based techniques to determine if a given record pair is indeed the same real world entity. If we search these terms, there are multiple articles around Deterministic Vs Probabilistic Matching, pitting one against the other. Since the approaches to solve deterministic matching are very different from the ones for probabilistic matching, these techniques are often viewed as contrary.

However, we have a different view.

As we dived deeper and looked at multiple datasets, we saw that in most cases, data contained multiple signals, some with trusted identifiers and most times without. Even when there are identifiers, they tend to be more than one. For example one system could have driver’s license id mandatory and passport number optional. While another system could have the mandatory rule reversed. A third system may have both of these ids as optional. If we match deterministically on passport and license in this case, we would only resolve some entities, not all. Worse, a person who has provided license id in the third system would only match with the first, and the record in the second system would go unmatched.

This has serious ramifications. In an ecommerce store, the inability to reconcile effectively would jeopardize the million dollar investment in recommender systems that consume the customer preferences. In an anti money laundering system, the ability to catch fraud is directly proportional to the quality of the matching. An insurer can not understand risk at a household or person level if records remain unresolved.

Thus, we need to solve for both determinstic and probabilistic matching at the same time. An either/or approach with respect to deterministic and probabilistic matching would lead to identifying records with identifiers as a different entity than those without. Why not use every single data point of the record? There is tremendous value to be unlocked by looking at the problem holistically. If a deterministic match can be found, use that. If not, use the other attributes. Essentially, weave them all together holistically to ensure that the probabilistic and deterministic matching records are resolved to a single entity.

The above is a simplistic take and there is a lot of software engineering involved under the hood to build this out. Effort nothwithstanding, we are convinced that it is the right approach. In fact we have gone all out while building this feature, with support for multiple combinations of attributes to make sure no record which has a match remains unmatched. Determistic and probabilistic matching are better together, and early Zingg Enterprise customers see an immediate jump in match accuracy.

Is this something you care about too? Hit a reply and lets spark a conversation!

]]> Sonal Goyal https://www.evalapply.org/posts/halting-ai/index.html https://www.evalapply.org/posts/halting-ai/index.html Wed, 13 Mar 2024 00:00:00 GMT Aditya Athalye meta riff ai intelligence_augmentation tools_for_thought https://ravidwivedi.in/posts/malaysia-trip/ https://ravidwivedi.in/posts/malaysia-trip/ Sat, 02 Mar 2024 13:59:59 GMT Introduction

In January of this year, my friend Snehal invited me to Vienna, Austria, but our plans were thwarted when Austria refused me a visa. So, I thought about traveling to a country that does not require a visa, and a few candidates came to mind, one of them being Malaysia which had recently waived visa requirements for Indian tourists. I also included Thailand, as it was visa-free, near to Malaysia, and is one of the most popular tourist destinations in the world.

I booked an AirAsia flight from Delhi, India, to Kuala Lumpur, Malaysia; a Malaysia Airlines flight from Kuala Lumpur to Bangkok; and an Air India ticket from Kuala Lumpur to Delhi.

Boarding from the Delhi Airport

On the 31st of January, before boarding my flight to Kuala Lumpur, I went to the AirAsia counter to obtain my boarding pass. During this process, the airline staff asked me a range of questions, from the usual - such as my purpose of visit, hotel bookings, and return ticket - to the ridiculous, including requests for invoices of hotel bookings, email confirmations of my return ticket, and the amount of currency I was carrying.

I didn’t have the invoices of my bookings as I didn’t pay for them in advance except for the first night, nor could I find the confirmation email by Air India. I was afraid that the airline staff might not give me the boarding pass, which is required to board the flight. I never had such experiences at airports in other countries.

Then I passed through the immigration where I was asked sensible questions such as my country of visit and return tickets. Following this, I went through security check, and proceeded to the flight’s boarding gate. Unfortunately, my flight was delayed by three hours. Even though AirAsia is a budget airline, I was surprised to find out that drinking water inside the plane was not free-of-cost, only hot drinking water was. It was a direct flight, which took 5 hours to reach Kuala Lumpur. I landed at Kuala Lumpur’s KLIA2 airport at around 09:00 hours local time (which is 2 hours 30 minutes ahead of Indian time) on the 1st of Feb.

Day 1: Arrival in Kuala Lumpur

After arrival at the Kuala Lumpur airport, I went through immigration where I was asked my purpose of visit, number of people accompanying me, number of days of stay, return/onward tickets and hotel bookings.

I had filled Malaysia Digital Arrival Card a day before my flight’s scheduled departure as per my travel agent’s advice, though the immigration officer didn’t ask for it. The immigration officer stamped my passport, after which I went through custom clearance. After clearing customs, I roamed around the airport and checked out SIM and internet plans, without buying.

Malaysia’s currency is the Malaysian ringgit, abbreviated as MYR. One MYR was equivalent to 18 Indian Rupees (INR). I exchanged Malaysian ringgit at a money exchange in my hometown.

My stay was booked at a hostel called Travel Hub Guesthouse in the Chinatown area of Kuala Lumpur, which was within walking distance of the Pasar Seni metro station, known as LRT station in local jargon. Every booking included a MYR 10 tourism tax per night. The hostel stay was 26.71 MYR plus 10 MYR (taxes), adding up to a total of 36.71 MYR (660 INR) per night.

KL Sentral serves as the city center and is located about 50 km from the airport. The rapid train from the airport to KL Sentral costs 55 MYR (approximately 1,000 INR), which I found to be quite expensive. Instead, I opted for the bus, which was a more economical choice at 15 MYR (approximately 270 INR). The bus took about an hour to reach KL Sentral and had comfortable seats. The well-maintained roads made my overall journey smooth and offered a glimpse of Kuala Lumpur city.

As soon as I checked in and entered my room, I met an Indian named Fletcher, who was also a tourist, and been in Kuala Lumpur for a couple of days, was planning to visit the National Museum and I joined him out of excitement for exploration, even though I was too tired after such a long trip and wanted to rest instead.

In the evening, I visited the National Museum (ticket was 5 MYR, equivalent to 90 INR) with Fletcher, and explored Little India, which had abundance of Indian food restaurants, especially by Tamils, making vegetarian food easily available. I came across a stall where I drank masala tea and sampled Mee Goreng. A quick search on the internet revealed that Mee Goreng is a dish unique to Indian immigrants in Malaysia and neighboring countries, but not found in India!

Day 2: Visiting Batu Caves and Petronas Towers

To stay with Fletcher, I extended my stay at the same hostel for one night. The next day (February 2nd), Fletcher went on a day trip to the Genting Highlands. Although I planned to join him, I couldn’t because his bus had no available bookings when we reached the KL Sentral station the next day. I noticed that at the KL Sentral station, bus tickets need to be paid for using a card or can be booked online through websites like RedBus. They had a cash option at the KL airport, but not at KL Sentral.

I took the opportunity to buy a local SIM card from the company CelcomDigi at the KL Sentral station for MYR 10, which included 5G internet up to 5 GB of usage. However, making calls required a recharge of MYR 5, which I didn’t include. At the bus ticket counter, I met a family from Delhi and joined them for a day trip to Batu Caves, which is a cave complex located on the outskirts of Kuala Lumpur.

To reach Batu Caves, we took the KTM Komuter train from KL Sentral station, which costs MYR 5.2 for a return journey. The train takes around 40 minutes to reach Batu Caves. Upon reaching there, we could immediately see the shrine of Murugan outside the Batu Caves. There were stairs leading to the caves. In order to take the stairs to the caves, make sure you cover your knees and shoulders. A woman from the family I went with was wearing shorts, so she had to buy a scarf worth MYR 15 to cover her knees for the entry. After climbing the stairs, we went inside the Temple Cave. This particular cave doesn’t have an entry fee, however some other caves (for example, the Ramayana Cave) have an entry fee. I didn’t go to the Ramayana Cave because I didn’t know about it.

The temperature inside the cave was cooler. We spent some time resting inside the cave. After we were done, we returned to KL Sentral, and went our separate ways. I came back to hostel and rested for some time.

My trip to Kuala Lumpur would be incomplete without a photo at the iconic Petronas Towers, which was my next destination. To get there from my hostel, I took the LRT to KLCC station and then walked to the photo point of the Petronas Towers, where I asked an Indonesian tourist for help in taking my pictures.

After roaming around at Petronas Towers, I gave a phone call to Fletcher who was on his way back to KL Sentral and we decided to meet there. We went to the same stall we had Mee Goreng the previous day and ordered Ghee Roast Dosa this time. We also had nice conversations with a family having Malaysian citizenship with Indian ancestry. Then, we went to another place to eat Roti Canai, which I had with dal. Interestingly, Roti Canai is another dish popularized by Indian immigrants in Southeast Asia, but it is not popular in India.

Day 3: Berjaya Times Square and Bukit Bintang

For the third day (3rd Feb), I could not extend my hostel booking as it was fully booked due to weekend. So, I booked another hostel by the name of The Manor by Mingle for two nights, 1 km far from my previous hostel, which was MYR 99.34 (1800 INR) for two nights, including MYR 10 tourism fee per night. This was expensive compared to other hostels I stayed in this trip, probably due to weekend. It had a swimming pool, which was of no use to me. Further, it also had laundry services, which I used. The charges were MYR 5 for washing, while MYR 3 for the dryer.

After checking-in and keeping my luggage into my room, I and Fletcher went to nearby shopping mall Berjaya Times Square, which was decorated in celebrations of the upcoming Chinese New Year on 8th Feb, due to which prices were greatly discounted. After roaming around and a bit of shopping, we went back to our respective hostels to take rest. At night, we went to Bukit Bintang, which is known for its nightlife and called the entertainment hub of the city.

Day 4: Genting Highlands

On the 4th of Feb, I took a solo day trip to Genting Highlands, a hill station located on the outskirts of Kuala Lumpur. To reach there, I took a bus from KL Sentral (return ticket was MYR 20), which dropped me at the bus terminal below Awana Skyway cable car station. I took a cable car to reach Genting Highlands from there (MYR 18 for a return ticket), which passes through misty air along with stunning views. I roamed around and did some shopping, buying three T-Shirts for myself. Furthermore, I also sampled Paneer Makhani with naan for MYR 41.8 at a restaurant.

There was not much to do here for me, as this place was popular for being the only legal place to gamble in Malaysia, nothing of my interest. Although the cable car ride had scenic views, I don’t think Genting Highlands was worth visiting. Sure, it’s a good place for a day trip from KL, but I had more time and should have instead gone to some other place like Cameron Highlands. Genting Highlands is for people staying for a couple of days in KL who don’t want to visit far-off places. I think my decision to visit Genting Highlands was a case of falling into the trap of herd mentality, as when I went to the bus counter a couple of days ago, I saw a big queue for Genting Highlands, which was the basis of my decision.

I took the return cable car to reach the bus terminal, from where I took the bus for KL Sentral. Then I went to meet Fletcher, who was leaving for Bangkok. We later met in Pattaya, Thailand, after a few days, which will be covered in the next post.

Days 5 and 6

Since the hostel Manor by Mingle was a bit expensive, I booked a cheaper hostel for last two nights in Bukit Bintang. I didn’t really do much on last two days. I roamed around and added some places on the OpenStreetMap. As I was living in Bukit Bintang, I bought some souveniers from there and tried a middle-eastern dessert named Kunafa, which was yummy. I also discovered a shop named ‘I Love KL Gifts’ which had souvenirs at a great price.

7th Feb: Malaysian Airlines to Bangkok

On the 7th of Feb, I took a Malaysian Airlines flight for Bangkok, which was scheduled to depart from KL Airport at 12:15 hours, but the departure got delayed by 2 hours, landing in Bangkok at around 15:00 hours local time (which is 1 hours 30 minutes ahead of Indian time). More details will be covered in the Thailand post.

Expenses

Category	Amount (INR)
Food + Accommodation + Travel in Malaysia	10,000
Delhi to Kuala Lumpur flight	13,000
Kuala Lumpur to Bangkok flight	10,000
Bangkok to Delhi flight	12,000

Learning for future trips

Malaysia has numerous places worthy of a visit: the Cameron Highlands (for its beautiful tea gardens), Malacca (due to its unique history and culture), Langkawi (for its white sand beaches) – and this list does not include the scenic spots from the part of Malaysia on the island of Borneo. However, I limited my trip to Kuala Lumpur and nearby places, which was a bad idea. After the trip, I realized that my mobility was reduced because I was carrying a trolley bag. Additionally, my trip to the Genting Highlands could have been replaced by a better alternative.

If you are like me and avoid taking a lot of taxis, preferring to walk or use public transport whenever possible, I highly advise you to cut back on luggage. Another instance where I could have benefited from this was with the airline ticket from Kuala Lumpur to Bangkok, which would have been much cheaper if I had carried less luggage (the tickets were 3,000-4,000 INR), compared to the ticket I bought for 10,000 INR.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/fix-internet-on-lineage-os/ https://ravidwivedi.in/posts/fix-internet-on-lineage-os/ Fri, 01 Mar 2024 09:04:08 GMT Network Settings -> Your SIM settings -> Access Point Names. Click on the ‘+’ symbol to add a new access point. In the Name section, you can input any name (e.g., test). In the APN section, enter www and save the settings. Check the screenshot below. APN settings screenshot. Notice the circled entries. Once you have added this new APN, ensure to select it from the list of available APNs. Following this configuration change, the issue got fixed. I hope this works for you :)]]> I have used LineageOS in many Android devices and face internet connectivity issues with mobile data. My mobile data never used to work properly in Xiaomi MI A2 on Lineage OS and my current phone OnePlus 9 Pro 5G. A few days ago, I met contrapunctus who has the same phone model with Lineage OS. He fixed this issue by comparing the settings of my phone with his.

In case, you are suffering from this issue, the following steps fixed the issue for me:

1. Navigate to Settings -> Network Settings -> Your SIM settings -> Access Point Names.
2. Click on the ‘+’ symbol to add a new access point.
3. In the Name section, you can input any name (e.g., test).
4. In the APN section, enter www and save the settings.

Check the screenshot below.

Once you have added this new APN, ensure to select it from the list of available APNs. Following this configuration change, the issue got fixed. I hope this works for you :)

]]> Ravi Dwivedi https://aryak.me/blog/07-encrypt-ext4-debian-install.html https://aryak.me/blog/07-encrypt-ext4-debian-install.html Thu, 29 Feb 2024 12:39:45 GMT Recently, I decided to encrypt the VPSes of Project Segfault, as it coincided with the migration of one of our servers, our EU node.

However, while moving our US node, we faced a few problems, in the fact that I didn’t want to wipe the disk, nor did I want to do some jank stuff like reinstalling debian and then replacing the files.

Therefore, my only solution came down to creating a new encrypted partition, copying the entire directory tree of the old partition to the new one via rsync, and then making grub and co. point to the new stuff.

So, in order to do this, you first need to shrink your existing partition so you can create the new one, which I did using GParted on the GParted LiveCD.

Past that, you have to create the new primary partition in the empty space created, which I did via CFDisk, since GParted requires you to format while creating a partition (from what I could see), and it doesn’t support creating LUKS partitions.

Additionally, since you can’t use the “good” PBKDF, argon2i(d) (which is more secure and gives faster speeds) with the current version of grub available on Debian 12, you have to move boot to a separate unencrypted partition. This can be done by merely creating a new ~512 MiB ext2 primary partition via GParted

Past this, I had to create the LUKS stuff. To do so, I ran the following commands:

# Format partition as LUKS encrypted
cryptsetup luksFormat --type luks2 --pbkdf argon2i /dev/DEVICE
# Open partition, and map it to /dev/mapper/crypt
cryptsetup luksOpen /dev/DEVICE crypt
# Overwrite the entire device with 0s, just to be a bit more secure
dd if=/dev/zero of=/dev/mapper/crypt status=progress bs=4096
# Create ext4 partition on the mapped device
mkfs.ext4 /dev/mapper/crypt

Then, to copy the content to the new partition, I used the following commands:

mkdir -p /mnt/{old,new}
mount /dev/OLD_DEVICE /mnt/old
mount /dev/mapper/crypt /mnt/new
mkdir -p /mnt/new/boot
mount /dev/BOOT_DEVICE /mnt/new/boot
rsync -av /mnt/old/* /mnt/new

After all the data is copied, I need to enter a chroot environment in order to configure a few more things. This is done through the following commands:

mount -t sysfs /sys /mnt/new/sys/
mount -t proc /proc /mnt/new/proc/
mount --rbind /dev /mnt/new/dev/
chroot /mnt/new

Inside the chroot environment, you need to first install some cryptsetup related stuff, and then update the fstab file:

apt install cryptsetup cryptsetup-initramfs

/etc/crypttab (the file which tells the system what encrypted partitions to mount):

crypt   UUID=UUID_OF_PARTITION_FROM_BLKID   none    luks,discard

/etc/fstab:

/dev/mapper/crypt   /   ext4    rw  0   1
/dev/BOOT_PARTITION /boot   ext2    rw  0   1

Past this, you need to reinstall grub:

grub-install /dev/DISK # for legacy BIOS, note this is the disk not the partition (ie. /dev/vda not /dev/vda1)
update-grub

And that should be it, once you reboot, you should boot into a password prompt, past which you can boot into your newly encrypted system!

PS: don’t forget to remove the old partition :D

]]> arya@projectsegfau.lt (Arya Kiran) 2024/02/29/4 https://www.prashanthudupa.com/asking-questions/ https://www.prashanthudupa.com/asking-questions/ Tue, 06 Feb 2024 03:14:39 GMT Prashanth Udupa Philosophy Unschooling https://aryak.me/blog/06-phone-webcam-scrcpy.html https://aryak.me/blog/06-phone-webcam-scrcpy.html Sat, 27 Jan 2024 12:39:45 GMT Recently, we decided to start streaming FOSS United Mumbai events to our Peertube instance, and hence we needed a camera.

Since we couldn’t procure a good camera that can capture text from the projector well, I started experimenting with other alternatives.

The first solution I tried was a generic IP camera based solution, which of course came with the latencies involved with network-based stuff and hence wasn’t suitable for this purpose.

Then, I tried droidcam, which uses ADB, but again, it had a paywall for anything over 480p.

Past this, my jugaad solution was to use scrcpy via USB ADB, open the normal camera app and then just capture part of the window with the content, but that came with the issue of only being able to use 4:3 aspect ratio, and potentially worse quality since liveview is rarely as good as actuals.

Later, I discovered that scrcpy can natively capture the phone camera since Android 12+ and Scrcpy 2.x+.

This came with the first problem, debian testing is still stuck on 1.x of scrcpy!

Note to self: maybe I should try porting a 2.x to Debian :D

As outlined in the docs, scrcpy can capture both front and back cameras, at all supported resolutions, and even the audio from the phone’s mic (which is useful considering how shitty the mics are on modern laptops these days) .

So at the end, I ended up with this command:

/usr/local/bin/scrcpy --video-source=camera --camera-id=0 --camera-size=3264x1836

Now, at this point, I just needed to run OBS under Xwayland, add it as an Xcomposite video capture, and start streaming!

But then, if you want to use it as a webcam, you just have to install v4l2loopback (and run modprobe v4l2loopback), and then click “Start Virtual Camera” instead of Start Streaming.

Do note though, you might have to change output type to scene under the settings section right next to the start virtual camera button.

But thats about it. This is still a bit jank, considering you need scrcpy and OBS both running in the background, but it generally does the job pretty well, and a good phone’s webcam is miles better than any other mid-range web-cam you can get.

]]> arya@projectsegfau.lt (Arya Kiran) 2024/01/27/6 https://kcsrk.info/ocaml/gdb/2024/01/20/gdb-ocaml/ https://kcsrk.info/ocaml/gdb/2024/01/20/gdb-ocaml/ Sat, 20 Jan 2024 15:16:00 GMT ._ where NNN is a randomly generated number. For the fib function, since it is under the file fib.ml, the module name is Fib. Since we can’t guess NNN, we use tab completion to help identify the function. (gdb) break camlFib.fib_ #press tab (gdb) break camlFib.fib_269 #269 happens to be the randomly generated number #on my machine. (gdb) Breakpoint 1 at 0x3d160: file fib.ml, line 1. You can also set a breakpoint using gdb’s file name and line number combination. Let’s set another break point at the main function, which is at line number 6 in fib.ml. (gdb) break fib.ml:6 Breakpoint 2 at 0x3d1d0: file fib.ml, line 6. Let’s run the program. (gdb) r Starting program: /home/kc/temp/fib.exe [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Breakpoint 2, camlFib.main_271 () at fib.ml:6 6 let main () = The program execution starts in the gdb session and we stop at the breakpoint installed at main. gdb has a nice TUI mode for stepping through the file. This can be activated with ctrl+x+a key combination, which should show a screen similar to the following. Notice that we can see both the breakpoints installed in this file. The current line is highlighted. Examining the stack You can step through the OCaml program with gdb commands n and s. After a few ns, you can examine the backtrace using the bt command. (gdb) bt #0 camlFib.fib_269 () at fib.ml:1 #1 0x00005555555911a1 in camlFib.fib_269 () at fib.ml:4 #2 0x00005555555911a1 in camlFib.fib_269 () at fib.ml:4 #3 0x00005555555911a1 in camlFib.fib_269 () at fib.ml:4 #4 0x00005555555911a1 in camlFib.fib_269 () at fib.ml:4 #5 0x00005555555911f1 in camlFib.main_271 () at fib.ml:7 #6 0x000055555559129a in camlFib.entry () at fib.ml:10 #7 0x000055555558eb0b in caml_program () #8 #9 0x00005555555dd306 in caml_startup_common (pooling=, argv=0x7fffffffe008) at runtime/startup_nat.c:132 #10 caml_startup_common (argv=0x7fffffffe008, pooling=) at runtime/startup_nat.c:88 #11 0x00005555555dd37f in caml_startup_exn (argv=) at runtime/startup_nat.c:139 #12 caml_startup (argv=) at runtime/startup_nat.c:144 #13 caml_main (argv=) at runtime/startup_nat.c:151 #14 0x000055555558e8f2 in main (argc=, argv=) at runtime/main.c:37 As you can see the backtrace includes the recursive calls to the fib function, the main function in fib.ml, followed by a number of functions from the OCaml runtime, and finally ending at the main function. Note that is a misnomer and is not an actual signal handler. OCaml 5 supports effect handlers with the help of runtime managed stack segments for the OCaml stack. There is also a single C stack that is used by all the fibers that run on a domain, our unit of parallelism. The represents a frame where the control switches between the C stack (managed by the OS) and the OCaml stack (managed by the OCaml runtime). The OCaml runtime marks these frames where the stack are split as signal handler frames so that gdb doesn’t complain about stack corruption; gdb expects stacks to grow down, which may not be true if the stack segments are in different parts of the memory address space. You will also find such frames between OCaml fibers (when using effect handlers) and when OCaml calls into the (C) runtime. You can find more details about the stack layout in the PLDI 2021 paper on OCaml effect handlers. Examining values There isn’t good support for examining OCaml values in gdb unlike C. That said, given the uniform value representation of OCaml, with a bit of information about the OCaml calling convention, we can start to examine the values. It is useful to note that OCaml 5.1.1 on x86 passes the first 10 arguments in registers. In particular, the first argument is in the register rax. So the argument to the fib function should be in the rax register. We also know that the argument to fib is an integer. OCaml uses 63-bit tagged integers (on 64-bit machines) with the least-significant bit is 1. Given a machine word or a register holding an OCaml integer, the integer value is obtained by right shifting the value by 1. Putting it all together, we can get the argument value of fib at the breakpoint at the entry to fib as follows: (gdb) p $rax >> 1 $2 = 12 Given that we’ve already stepped through the program several times, the current call for me corresponds to fib(12). Let’s see what’s the next argument by continuing the program until we hit the breakpoint again. (gdb) c Continuing. Breakpoint 1, camlFib.fib_269 () at fib.ml:1 (gdb) p $rax >> 1 $3 = 10 Observe that this corresponds to the recursive call fib(10), which must mean that the RHS recursive call is the one being invoked. Note that the evaluation order of arguments in OCaml is unspecified. The 5.1.1 implementation does right-to-left evaluation of arguments (to the (+) function in this case), which can be confirmed with the following program: $ cat eval_order.ml let _ = (print_endline "hello"; 0) + (print_endline "world"; 1) $ ocamlopt.opt -g -o eval_order.exe eval_order.ml $ ./eval_order.exe world hello Advanced printing As you can observe, examining values this way is cumbersome. The OCaml compiler distribution has some rudimentary scripts to make it easier to examine OCaml values in gdb. Note that this was developed by OCaml maintainers to develop the compiler, and was not designed to serve end user needs. That said, let’s dive in. Since we are on OCaml 5.1.1, let’s check out the source code for 5.1.1 first. # I'm in ~/repos directory on my machine *) $ git clone https://github.com/ocaml/ocaml --branch 5.1.1 Let’s start a new gdb session, load the gdb script and get to the desired breakpoint. $ gdb ./fib.exe (gdb) source ~/repos/ocaml/tools/gdb_ocamlrun.py (gdb) break fib.ml:1 Breakpoint 1 at 0x3d160: file fib.ml, line 1. (gdb) r Starting program: /home/kc/temp/fib.exe [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Breakpoint 1, camlFib.fib_269 () at fib.ml:1 1 let rec fib n = As earlier, the first argument is in rax register. We can examine the value now with the help of the script. (gdb) p (value)$rax $1 = I(20) value is the type of OCaml values defined in OCaml runtime. The script tools/gdb_ocamlrun.py installs a pretty printer for the values of type value. Here, it prints that the argument is the integer 20. We can also print other kinds of OCaml values. In order to illustrate this, consider the following program: $ cat test_blocks.ml (* test_blocks.ml *) type t = {s : string; i : int} let main a b = print_endline "Hello, world!"; print_endline a; print_endline b.s let _ = main "foo" {s = "bar"; i = 42} Let’s compile, start a gdb session and break at the main function. $ ocamlopt -g -o test_blocks.exe test_blocks.ml $ gdb ./test_blocks.exe (gdb) break camlTest_blocks.main_272 Breakpoint 1 at 0x16ed0: file test_blocks.ml, line 5. (gdb) r Starting program: /home/kc/temp/test_blocks.exe [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Breakpoint 1, camlTest_blocks.main_272 () at test_blocks.ml:5 5 let main a b = (gdb) source ~/repos/ocaml/tools/gdb_ocamlrun.py Let’s examine the two arguments to main. (gdb) p (value)$rax $1 = String_tag("foo", NOT_MARKABLE) The first argument is a string “foo”. NOT_MARKABLE is one of the GC colours used by OCaml 5, and represents objects that are not traced by the mark-and-sweep (major) GC. The string happens to be allocated in the data section of the address space, and is not traced by the GC. (gdb) info symbol $rax camlTest_blocks.4 in section .data of /home/kc/temp/test_blocks.exe Let’s examine the second argument. (gdb) p (value)$rbx $2 = Block(0, wosize=2, NOT_MARKABLE) The second argument, which is passed in the register rbx, is a record with two fields. Hence, the pretty printer says that it is a block with 2 fields. We can print both values using gdb’s support for printing a range of values. (gdb) p *(value*)$rbx@2 $3 = {String_tag("bar", NOT_MARKABLE), I(42)} We cast rbx to an array of values and print the first two fields in the array. This shows that the fields are the string “bar” and integer 42. More for later There is a lot more to be said about debugging OCaml programs using gdb. We shall see them in subsequent posts if there is interest.]]> A number of folks who regularly use OCaml were surprised to learn that you can reasonably debug OCaml programs using gdb. The aim of the post is to show the first steps in using gdb on OCaml programs.

Let’s consider the following program:

(* fib.ml *)
let rec fib n = 
  if n = 0 then 0
  else if n = 1 then 1
  else fib (n-1) + fib (n-2)

let main () = 
  let r = fib 20 in 
  Printf.printf "fib(20) = %d" r

let _ = main ()

Let’s compile this program. I’m using OCaml version 5.1.1.

$ ocamlopt --version
5.1.1
$ ocamlopt -g -o fib.exe fib.ml
$ $ ./fib.exe 20
fib(20) = 6765

As you can see, the program prints the 20th Fibonacci number. Let’s examine this program under gdb. Before we venture any further, I highly recommend watching this 15-minute video that shows a number of gdb tricks. Let’s start a gdb session.

$ gdb ./fib.exe

Setting breakpoints

Let’s set a break point at the fib function. When OCaml functions are compiled, their names are mangled. OCaml 5.1.1 uses the following mangling scheme caml<MODULE_NAME>.<FUNCTION_NAME>_<NNN> where NNN is a randomly generated number. For the fib function, since it is under the file fib.ml, the module name is Fib. Since we can’t guess NNN, we use tab completion to help identify the function.

(gdb) break camlFib.fib_ #press tab
(gdb) break camlFib.fib_269 #269 happens to be the randomly generated number
                            #on my machine.
(gdb) Breakpoint 1 at 0x3d160: file fib.ml, line 1.

You can also set a breakpoint using gdb’s file name and line number combination. Let’s set another break point at the main function, which is at line number 6 in fib.ml.

(gdb) break fib.ml:6
Breakpoint 2 at 0x3d1d0: file fib.ml, line 6.

Let’s run the program.

(gdb) r
Starting program: /home/kc/temp/fib.exe 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 2, camlFib.main_271 () at fib.ml:6
6       let main () =

The program execution starts in the gdb session and we stop at the breakpoint installed at main. gdb has a nice TUI mode for stepping through the file. This can be activated with ctrl+x+a key combination, which should show a screen similar to the following.

Notice that we can see both the breakpoints installed in this file. The current line is highlighted.

Examining the stack

You can step through the OCaml program with gdb commands n and s. After a few ns, you can examine the backtrace using the bt command.

(gdb) bt
#0  camlFib.fib_269 () at fib.ml:1
#1  0x00005555555911a1 in camlFib.fib_269 () at fib.ml:4
#2  0x00005555555911a1 in camlFib.fib_269 () at fib.ml:4
#3  0x00005555555911a1 in camlFib.fib_269 () at fib.ml:4
#4  0x00005555555911a1 in camlFib.fib_269 () at fib.ml:4
#5  0x00005555555911f1 in camlFib.main_271 () at fib.ml:7
#6  0x000055555559129a in camlFib.entry () at fib.ml:10
#7  0x000055555558eb0b in caml_program ()
#8  <signal handler called>
#9  0x00005555555dd306 in caml_startup_common (pooling=<optimised out>, argv=0x7fffffffe008) at runtime/startup_nat.c:132
#10 caml_startup_common (argv=0x7fffffffe008, pooling=<optimised out>) at runtime/startup_nat.c:88
#11 0x00005555555dd37f in caml_startup_exn (argv=<optimised out>) at runtime/startup_nat.c:139
#12 caml_startup (argv=<optimised out>) at runtime/startup_nat.c:144
#13 caml_main (argv=<optimised out>) at runtime/startup_nat.c:151
#14 0x000055555558e8f2 in main (argc=<optimised out>, argv=<optimised out>) at runtime/main.c:37

As you can see the backtrace includes the recursive calls to the fib function, the main function in fib.ml, followed by a number of functions from the OCaml runtime, and finally ending at the main function.

Note that <signal handler called> is a misnomer and is not an actual signal handler. OCaml 5 supports effect handlers with the help of runtime managed stack segments for the OCaml stack. There is also a single C stack that is used by all the fibers that run on a domain, our unit of parallelism. The <signal handler called> represents a frame where the control switches between the C stack (managed by the OS) and the OCaml stack (managed by the OCaml runtime). The OCaml runtime marks these frames where the stack are split as signal handler frames so that gdb doesn’t complain about stack corruption; gdb expects stacks to grow down, which may not be true if the stack segments are in different parts of the memory address space. You will also find such <signal handler called> frames between OCaml fibers (when using effect handlers) and when OCaml calls into the (C) runtime. You can find more details about the stack layout in the PLDI 2021 paper on OCaml effect handlers.

Examining values

There isn’t good support for examining OCaml values in gdb unlike C. That said, given the uniform value representation of OCaml, with a bit of information about the OCaml calling convention, we can start to examine the values. It is useful to note that OCaml 5.1.1 on x86 passes the first 10 arguments in registers. In particular, the first argument is in the register rax. So the argument to the fib function should be in the rax register. We also know that the argument to fib is an integer. OCaml uses 63-bit tagged integers (on 64-bit machines) with the least-significant bit is 1. Given a machine word or a register holding an OCaml integer, the integer value is obtained by right shifting the value by 1.

Putting it all together, we can get the argument value of fib at the breakpoint at the entry to fib as follows:

(gdb) p $rax >> 1
$2 = 12

Given that we’ve already stepped through the program several times, the current call for me corresponds to fib(12). Let’s see what’s the next argument by continuing the program until we hit the breakpoint again.

(gdb) c
Continuing.

Breakpoint 1, camlFib.fib_269 () at fib.ml:1
(gdb) p $rax >> 1
$3 = 10

Observe that this corresponds to the recursive call fib(10), which must mean that the RHS recursive call is the one being invoked. Note that the evaluation order of arguments in OCaml is unspecified. The 5.1.1 implementation does right-to-left evaluation of arguments (to the (+) function in this case), which can be confirmed with the following program:

$ cat eval_order.ml
let _ =
  (print_endline "hello"; 0) + (print_endline "world"; 1)
$ ocamlopt.opt -g -o eval_order.exe eval_order.ml
$ ./eval_order.exe
world
hello

Advanced printing

As you can observe, examining values this way is cumbersome. The OCaml compiler distribution has some rudimentary scripts to make it easier to examine OCaml values in gdb. Note that this was developed by OCaml maintainers to develop the compiler, and was not designed to serve end user needs. That said, let’s dive in.

Since we are on OCaml 5.1.1, let’s check out the source code for 5.1.1 first.

# I'm in ~/repos directory on my machine *)
$ git clone https://github.com/ocaml/ocaml --branch 5.1.1

Let’s start a new gdb session, load the gdb script and get to the desired breakpoint.

$ gdb ./fib.exe
(gdb) source ~/repos/ocaml/tools/gdb_ocamlrun.py
(gdb) break fib.ml:1
Breakpoint 1 at 0x3d160: file fib.ml, line 1.
(gdb) r
Starting program: /home/kc/temp/fib.exe 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, camlFib.fib_269 () at fib.ml:1
1       let rec fib n =

As earlier, the first argument is in rax register. We can examine the value now with the help of the script.

(gdb) p (value)$rax
$1 = I(20)

value is the type of OCaml values defined in OCaml runtime. The script tools/gdb_ocamlrun.py installs a pretty printer for the values of type value. Here, it prints that the argument is the integer 20.

We can also print other kinds of OCaml values. In order to illustrate this, consider the following program:

$ cat test_blocks.ml
(* test_blocks.ml *)

type t = {s : string; i : int}

let main a b =
  print_endline "Hello, world!";
  print_endline a;
  print_endline b.s

let _ = main "foo" {s = "bar"; i = 42}

Let’s compile, start a gdb session and break at the main function.

$ ocamlopt -g -o test_blocks.exe test_blocks.ml                                                                                                               
$ gdb ./test_blocks.exe
(gdb) break camlTest_blocks.main_272 
Breakpoint 1 at 0x16ed0: file test_blocks.ml, line 5.
(gdb) r
Starting program: /home/kc/temp/test_blocks.exe 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, camlTest_blocks.main_272 () at test_blocks.ml:5
5       let main a b =
(gdb) source ~/repos/ocaml/tools/gdb_ocamlrun.py 

Let’s examine the two arguments to main.

(gdb) p (value)$rax
$1 = String_tag("foo", NOT_MARKABLE)

The first argument is a string “foo”. NOT_MARKABLE is one of the GC colours used by OCaml 5, and represents objects that are not traced by the mark-and-sweep (major) GC. The string happens to be allocated in the data section of the address space, and is not traced by the GC.

(gdb) info symbol $rax
camlTest_blocks.4 in section .data of /home/kc/temp/test_blocks.exe

Let’s examine the second argument.

(gdb) p (value)$rbx
$2 = Block(0, wosize=2, NOT_MARKABLE)

The second argument, which is passed in the register rbx, is a record with two fields. Hence, the pretty printer says that it is a block with 2 fields. We can print both values using gdb’s support for printing a range of values.

(gdb) p *(value*)$rbx@2                                                                 
$3 = {String_tag("bar", NOT_MARKABLE), I(42)}

We cast rbx to an array of values and print the first two fields in the array. This shows that the fields are the string “bar” and integer 42.

More for later

There is a lot more to be said about debugging OCaml programs using gdb. We shall see them in subsequent posts if there is interest.

]]> KC Sivaramakrishnan https://mrkaran.dev/posts/1brc/ https://mrkaran.dev/posts/1brc/ Wed, 10 Jan 2024 00:00:00 GMT max { max = t } sum += t } mean := sum / float64(len(temps)) results[station] = Stats{Min: min, Mean: mean, Max: max} } // Sort the stations and format the output. var stations []string for station := range results { stations = append(stations, station) } sort.Strings(stations) fmt.Print("{") for i, station := range stations { r := results[station] fmt.Printf("%s=%.1f/%.1f/%.1f", station, r.Min, r.Mean, r.Max) if i < len(stations)-1 { fmt.Print(", ") } } fmt.Println("}") } On running the above program, we get the following output: {Chihuahua=20.3/20.3/20.3, Dikson=-3.7/-3.7/-3.7, Dodoma=20.3/20.3/20.3, Kampala=50.8/50.8/50.8, Mek'ele=13.3/13.3/13.3, Ngaoundéré=24.2/24.2/24.2, San Diego=7.1/7.1/7.1, Singapore=14.4/14.4/14.4, Toronto=12.7/12.7/12.7, Wrocław=12.6/12.6/12.6} This approach works well for small, simple files. However, there are certain restrictions: It reads the file line by line using a scanner. Reading and processing a billion rows is time-consuming. Each operation, even if small, adds up when repeated a billion times. This includes string splitting, type conversion, error checking, and appending to a slice. Additionally, we need to consider the potential of hitting the max Disk IOPS limit if we perform too many file operations per second. Before we proceed to optimize this further, let’s establish a baseline performance of 100 million lines first: $ wc -l measurements.txt 100000000 measurements.txt $ time go run main.go go run main.go 18.44s user 0.83s system 100% cpu 19.135 total Baseline: It takes approximately 19s to read and calculate stats from 100 mn lines. There’s a lot of room to optimize it further, let’s go through them one by one. Iteration 1: Producer-Consumer Pattern# The concept involves reading multiple lines simultaneously in the producer Goroutine and then dispatching these batches to worker Goroutines. We can establish a worker pool to implement a producer-consumer pattern. Producers read lines from the file and send them to a channel. Consumers retrieve lines from the channel, parse the data, and calculate the minimum, mean, and maximum temperatures for each station. func main() { numWorkers := runtime.NumCPU() runtime.GOMAXPROCS(numWorkers) linesChan := make(chan string, 1000000) resultsChan := make(chan map[string]Stats, numWorkers) // Start worker goroutines var wg sync.WaitGroup for i := 0; i < numWorkers; i++ { wg.Add(1) go worker(linesChan, resultsChan, &wg) } // Read the file and send lines to the workers go func() { file, err := os.Open(measurementsFile) if err != nil { panic(err) } defer file.Close() scanner := bufio.NewScanner(file) for scanner.Scan() { linesChan <- scanner.Text() } close(linesChan) }() // Collect results from workers wg.Wait() close(resultsChan) // Aggregate results finalResults := make(map[string]Stats) for workerResult := range resultsChan { for station, stats := range workerResult { finalStats := finalResults[station] finalStats.Min = min(finalStats.Min, stats.Min) finalStats.Max = max(finalStats.Max, stats.Max) finalStats.Mean = (finalStats.Mean*float64(finalStats.Count) + stats.Mean*float64(stats.Count)) / float64(finalStats.Count+stats.Count) finalStats.Count += stats.Count finalResults[station] = finalStats } } // Print results printStats(finalResults) } func worker(linesChan <-chan string, resultsChan chan<- map[string]Stats, wg *sync.WaitGroup) { defer wg.Done() stationStats := make(map[string]Stats) for line := range linesChan { parts := strings.Split(line, ";") temp, err := strconv.ParseFloat(parts[1], 64) if err != nil { continue } stats := stationStats[parts[0]] stats.Count++ stats.Min = min(stats.Min, temp) stats.Max = max(stats.Max, temp) stats.Mean += (temp - stats.Mean) / float64(stats.Count) stationStats[parts[0]] = stats } resultsChan <- stationStats } func min(a, b float64) float64 { if a == 0 || a > b { return b } return a } func max(a, b float64) float64 { if a < b { return b } return a } func printStats(statsMap map[string]Stats) { var stations []string for station := range statsMap { stations = append(stations, station) } sort.Strings(stations) fmt.Print("{") for i, station := range stations { stats := statsMap[station] fmt.Printf("%s=%.1f/%.1f/%.1f", station, stats.Min, stats.Mean, stats.Max) if i < len(stations)-1 { fmt.Print(", ") } } fmt.Println("}") } Results# The concurrent version, unexpectedly, resulted in almost a 3x decrease in performance. go run main.go 84.15s user 101.34s system 342% cpu 54.225 total Where did we go wrong? This is a classic case where the overhead of concurrency mechanisms outweighs their benefits. In our current implementation, each line is sent to the channel individually, which is likely less efficient than batching lines for processing. This means that for a file with a large number of lines, there will be an equally large number of channel send operations. Each channel operation involves locking and unlocking, which can be costly, especially in a high-frequency context. Iteration 2: Batch processing of lines# In this version we are Batching the lines before sending to the worker which will significantly reduce the overhead of channel communication. Batch Processing: Each batch contains batchSize lines. This reduces the frequency of channel operations (both sending and receiving), as well as the overhead associated with these operations. Efficient Worker Utilization: With batch processing, each worker goroutine spends more time processing data and less time interacting with channels. This reduces the overhead of context switching and synchronization, making the processing more efficient. const ( batchSize = 1000000 // Number of lines per batch ) // ... scanner := bufio.NewScanner(file) var batch []string for scanner.Scan() { batch = append(batch, scanner.Text()) if len(batch) >= batchSize { batchesChan <- batch batch = nil // Start a new batch } } // Send any remaining lines in the last batch if len(batch) > 0 { batchesChan <- batch } close(batchesChan) // ... func worker(batchesChan <-chan []string, resultsChan chan<- map[string]Stats, wg *sync.WaitGroup) { defer wg.Done() stationStats := make(map[string]Stats) for batch := range batchesChan { for _, line := range batch { // Process the line ... } } resultsChan <- stationStats } // ... Results# The improvement from iteration 2 to iteration 3 is quite remarkable, thanks to efficiently batching the lines together and reducing the number of channel ops. go run main.go 30.02s user 0.67s system 476% cpu 6.442 total So far, we’ve reduced the time to about 6.5s which is a great start and improvement of our baseline version of 19s. However, we’re making quite a few extra memory allocations and the focus of next iteration should be to reduce that. Iteration 3: Reducing memory allocations# A batch slice is pre-allocated with a capacity of batchSize and reused for each batch of lines. After sending a batch to the channel, the slice is reset to zero length (batch = batch[:0]), but the underlying array is retained and reused. // Read the file and send batches of lines to the workers go func() { file, err := os.Open(measurementsFile) if err != nil { panic(err) } defer file.Close() scanner := bufio.NewScanner(file) batch := make([]string, 0, batchSize) // Pre-allocate with capacity for scanner.Scan() { line := scanner.Text() // Reuse the batch slice by appending to it until it reaches the batch size batch = append(batch, line) if len(batch) >= batchSize { batchesChan <- batch batch = batch[:0] // Reset the slice without allocating new memory } } // Send any remaining lines in the last batch if len(batch) > 0 { batchesChan <- batch } close(batchesChan) }() Results# Down to 5.3s! go run main.go 25.43s user 0.53s system 485% cpu 5.346 total Iteration 3 (cont): Further reducing memory allocations# Avoiding strings.Split: Instead of using strings.Split, which allocates a new slice for each line, we can use strings.Index to find the delimiter and manually slice the string. strings.Split typically creates a new slice for each split part, leading to more memory usage and subsequent GC overhead. for batch := range batchesChan { for _, line := range batch { delimiterIndex := strings.Index(line, ";") if delimiterIndex == -1 { continue // Delimiter not found, skip this line } station := line[:delimiterIndex] tempStr := line[delimiterIndex+1:] temp, err := strconv.ParseFloat(tempStr, 64) if err != nil { continue // Invalid temperature value, skip this line } stats := stationStats[station] stats.Count++ stats.Min = min(stats.Min, temp) stats.Max = max(stats.Max, temp) stats.Mean += (temp - stats.Mean) / float64(stats.Count) stationStats[station] = stats } } Results# The time has further decreased from 5.3s to 4.8s with these changes. go run main.go 15.69s user 0.44s system 332% cpu 4.853 total Iteration 4: Read file in chunks# In this version, the file is read in chunks, and each chunk is processed to ensure it contains complete lines. The processChunk function is used to separate valid data from leftover data in each chunk. Chunk size can be controlled with command line args as well. func main() { // .... const chunkSize = 256 * 1024 // 256 KB buf := make([]byte, chunkSize) leftover := make([]byte, 0, chunkSize) go func() { for { bytesRead, err := file.Read(buf) if bytesRead > 0 { // Copy the chunk to a new slice, because the // buffer will be reused in the next iteration. chunk := make([]byte, bytesRead) copy(chunk, buf[:bytesRead]) // Process the chunk. The returned leftover will be processed in the next iteration. validChunk, newLeftover := processChunk(chunk, leftover) leftover = newLeftover // Send the valid chunk to the processing goroutine. if len(validChunk) > 0 { wg.Add(1) go processChunkData(validChunk, resultsChan, &wg) } } if err != nil { break } } wg.Wait() close(resultsChan) }() // ... } func processChunk(chunk, leftover []byte) (validChunk, newLeftover []byte) { firstNewline := -1 lastNewline := -1 // Find the first and last newline in the chunk. for i, b := range chunk { if b == '\n' { if firstNewline == -1 { firstNewline = i } lastNewline = i } } if firstNewline != -1 { validChunk = append(leftover, chunk[:lastNewline+1]...) newLeftover = make([]byte, len(chunk[lastNewline+1:])) copy(newLeftover, chunk[lastNewline+1:]) } else { newLeftover = append(leftover, chunk...) } return validChunk, newLeftover } func processChunkData(chunk []byte, resultsChan chan<- map[string]Stats, wg *sync.WaitGroup) { defer wg.Done() stationStats := make(map[string]Stats) scanner := bufio.NewScanner(strings.NewReader(string(chunk))) for scanner.Scan() { line := scanner.Text() // Find the index of the delimiter delimiterIndex := strings.Index(line, ";") if delimiterIndex == -1 { continue // Delimiter not found, skip this line } // Extract the station name and temperature string station := line[:delimiterIndex] tempStr := line[delimiterIndex+1:] // Convert the temperature string to a float temp, err := strconv.ParseFloat(tempStr, 64) if err != nil { continue // Invalid temperature value, skip this line } // Update the statistics for the station stats, exists := stationStats[station] if !exists { stats = Stats{Min: temp, Max: temp} } stats.Count++ stats.Min = min(stats.Min, temp) stats.Max = max(stats.Max, temp) stats.Mean += (temp - stats.Mean) / float64(stats.Count) stationStats[station] = stats } // Send the computed stats to resultsChan resultsChan <- stationStats } In addition to this, I moved the aggregateStats to a separate Goroutine as well: aggWg.Add(1) finalResults := make(map[string]Stats) // Start a separate goroutine for aggregation go func() { defer aggWg.Done() for workerResult := range resultsChan { for station, stats := range workerResult { finalStats, exists := finalResults[station] if !exists { finalResults[station] = stats continue } finalStats.Min = min(finalStats.Min, stats.Min) finalStats.Max = max(finalStats.Max, stats.Max) totalCount := finalStats.Count + stats.Count finalStats.Mean = (finalStats.Mean*float64(finalStats.Count) + stats.Mean*float64(stats.Count)) / float64(totalCount) finalStats.Count = totalCount finalResults[station] = finalStats } } }() Results# We’re down from 4.8s to just 2.1s to read/parse/process 100mn lines! ./bin/1brc.bin --file=input.txt --chunksize=1048576 17.58s user 0.77s system 837% cpu 2.190 total Summary# Basic File Reading and Parsing (Baseline): Time: 19s (baseline). Key Change: Sequentially reading and processing each line. Speedup: N/A (baseline). Producer-Consumer Pattern: Time: 54.225s. Key Change: Implemented concurrent line processing with producer-consumer pattern. Speedup: -185% (slower than baseline). Batch Processing of Lines: Time: 6.442s. Key Change: Batched lines before processing, reducing channel communication. Speedup: +66% (compared to baseline). Reducing Memory Allocations - Iteration 1: Time: 5.346s. Key Change: Reused batch slices and reduced memory allocations. Speedup: +72% (compared to baseline). Reducing Memory Allocations - Iteration 2 (Avoiding strings.Split): Time: 4.853s. Key Change: Replaced strings.Split with manual slicing for efficiency. Speedup: +75% (compared to baseline). Read File in Chunks: Time: 2.190s. Key Change: Processed file in chunks and optimized aggregation. Speedup: +87% (compared to baseline). Final Run# I’m quite satisfied with the final version for now. We can now proceed to test it with 1 billion lines. However it’s evidently CPU-bound, as we spawn N workers for N CPUs. I experimented with different chunk sizes, and here are the results from each run: Chunk SizeTime 512.00 KB23.756s 1.00 MB21.798s 16.00 MB20.693s 32.00 MB19.501s Tweaking the chunk size doesn’t significantly impact performance, as processing larger chunks takes longer. TL;DR: On an average and with multiple runs it takes approx 20s with the final iteration for 1bn lines. Checkout the full code on my GitHub. Potential Improvements# This project was not only fun but also a great opportunity to revisit and refine many Go concepts. There are several ideas to contemplate for further improving this version’s timings: I haven’t yet considered using mmap, but I believe it could substantially speed things up. To delve even deeper, custom line parsing functions, especially for converting string to float64, could offer improvements. Employing custom hashing functions (perhaps FnV) might aid in faster map lookups. Fin!]]> Earlier this week, I had stumbled upon 1brc, which presents a fun task: loading a huge text file (1 billion lines) in Java as quickly as possible.

The One Billion Row Challenge (1BRC) is a fun exploration of how far modern Java can be pushed for aggregating one billion rows from a text file. Utilize all your virtual threads, leverage SIMD, optimize your GC, or employ any other technique to create the fastest implementation for this task!

The challenge is mainly about Java, but I thought to do the same in my preferred language: Go. This post is about how I did several iterations to my Go program to reduce the time and discuss the main techniques used in each iteration to make it faster.

I was able to create a solution which takes ~20s to read, parse and calculate stats for 1bn lines on my Apple M2 (10 vCPU, 32GB RAM). There are some insane solutions that people have come up with, be sure to check out GitHub Discussions to go through them!

Prerequisites#

To generate the text file for these measurements, follow the steps outlined here.

After running the commands, I have a measurements.txt on my file system:

Example output after running the commands:

➜  1brc-go git:(main) du -sh measurements.txt
 13G	measurements.txt
➜  1brc-go git:(main) tail measurements.txt
Mek'ele;13.3
Kampala;50.8
Dikson;-3.7
Dodoma;20.3
San Diego;7.1
Chihuahua;20.3
Ngaoundéré;24.2
Toronto;12.7
Wrocław;12.6
Singapore;14.4

Ultra minimalistic example of reading a file#

Let’s take a look at a basic Go code to read and parse the above file. We’ll also calculate stats on the fly.

package main

import (
	"bufio"
	"fmt"
	"os"
	"sort"
	"strconv"
	"strings"
)

type Measurement struct {
	Station string
	Temp    float64
}

type Stats struct {
	Min, Mean, Max float64
}

func main() {
	// Open the file.
	file, err := os.Open("measurements.txt")
	if err != nil {
		panic(err)
	}
	defer file.Close()

	// Map to hold the temperatures for each station.
	stationTemps := make(map[string][]float64)

	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		// Parse each line into a Measurement struct.
		parts := strings.Split(scanner.Text(), ";")
		temp, _ := strconv.ParseFloat(parts[1], 64)
		stationTemps[parts[0]] = append(stationTemps[parts[0]], temp)
	}

	// Calculate min, mean, and max for each station.
	results := make(map[string]Stats)
	for station, temps := range stationTemps {
		min, max, sum := temps[0], temps[0], 0.0
		for _, t := range temps {
			if t < min {
				min = t
			}
			if t > max {
				max = t
			}
			sum += t
		}
		mean := sum / float64(len(temps))
		results[station] = Stats{Min: min, Mean: mean, Max: max}
	}

	// Sort the stations and format the output.
	var stations []string
	for station := range results {
		stations = append(stations, station)
	}
	sort.Strings(stations)

	fmt.Print("{")
	for i, station := range stations {
		r := results[station]
		fmt.Printf("%s=%.1f/%.1f/%.1f", station, r.Min, r.Mean, r.Max)
		if i < len(stations)-1 {
			fmt.Print(", ")
		}
	}
	fmt.Println("}")
}

On running the above program, we get the following output:

{Chihuahua=20.3/20.3/20.3, Dikson=-3.7/-3.7/-3.7, Dodoma=20.3/20.3/20.3, Kampala=50.8/50.8/50.8, Mek'ele=13.3/13.3/13.3, Ngaoundéré=24.2/24.2/24.2, San Diego=7.1/7.1/7.1, Singapore=14.4/14.4/14.4, Toronto=12.7/12.7/12.7, Wrocław=12.6/12.6/12.6}

This approach works well for small, simple files. However, there are certain restrictions:

• It reads the file line by line using a scanner. Reading and processing a billion rows is time-consuming.
• Each operation, even if small, adds up when repeated a billion times. This includes string splitting, type conversion, error checking, and appending to a slice.
• Additionally, we need to consider the potential of hitting the max Disk IOPS limit if we perform too many file operations per second.

Before we proceed to optimize this further, let’s establish a baseline performance of 100 million lines first:

$ wc -l measurements.txt
  100000000 measurements.txt
$ time go run main.go
  go run main.go  18.44s user 0.83s system 100% cpu 19.135 total

Baseline: It takes approximately 19s to read and calculate stats from 100 mn lines.

There’s a lot of room to optimize it further, let’s go through them one by one.

Iteration 1: Producer-Consumer Pattern#

The concept involves reading multiple lines simultaneously in the producer Goroutine and then dispatching these batches to worker Goroutines. We can establish a worker pool to implement a producer-consumer pattern. Producers read lines from the file and send them to a channel. Consumers retrieve lines from the channel, parse the data, and calculate the minimum, mean, and maximum temperatures for each station.

func main() {
	numWorkers := runtime.NumCPU()
	runtime.GOMAXPROCS(numWorkers)

	linesChan := make(chan string, 1000000)
	resultsChan := make(chan map[string]Stats, numWorkers)

	// Start worker goroutines
	var wg sync.WaitGroup
	for i := 0; i < numWorkers; i++ {
		wg.Add(1)
		go worker(linesChan, resultsChan, &wg)
	}

	// Read the file and send lines to the workers
	go func() {
		file, err := os.Open(measurementsFile)
		if err != nil {
			panic(err)
		}
		defer file.Close()

		scanner := bufio.NewScanner(file)
		for scanner.Scan() {
			linesChan <- scanner.Text()
		}
		close(linesChan)
	}()

	// Collect results from workers
	wg.Wait()
	close(resultsChan)

	// Aggregate results
	finalResults := make(map[string]Stats)
	for workerResult := range resultsChan {
		for station, stats := range workerResult {
			finalStats := finalResults[station]
			finalStats.Min = min(finalStats.Min, stats.Min)
			finalStats.Max = max(finalStats.Max, stats.Max)
			finalStats.Mean = (finalStats.Mean*float64(finalStats.Count) + stats.Mean*float64(stats.Count)) / float64(finalStats.Count+stats.Count)
			finalStats.Count += stats.Count
			finalResults[station] = finalStats
		}
	}

	// Print results
	printStats(finalResults)
}

func worker(linesChan <-chan string, resultsChan chan<- map[string]Stats, wg *sync.WaitGroup) {
	defer wg.Done()

	stationStats := make(map[string]Stats)
	for line := range linesChan {
		parts := strings.Split(line, ";")
		temp, err := strconv.ParseFloat(parts[1], 64)
		if err != nil {
			continue
		}

		stats := stationStats[parts[0]]
		stats.Count++
		stats.Min = min(stats.Min, temp)
		stats.Max = max(stats.Max, temp)
		stats.Mean += (temp - stats.Mean) / float64(stats.Count)
		stationStats[parts[0]] = stats
	}

	resultsChan <- stationStats
}

func min(a, b float64) float64 {
	if a == 0 || a > b {
		return b
	}
	return a
}

func max(a, b float64) float64 {
	if a < b {
		return b
	}
	return a
}

func printStats(statsMap map[string]Stats) {
	var stations []string
	for station := range statsMap {
		stations = append(stations, station)
	}
	sort.Strings(stations)

	fmt.Print("{")
	for i, station := range stations {
		stats := statsMap[station]
		fmt.Printf("%s=%.1f/%.1f/%.1f", station, stats.Min, stats.Mean, stats.Max)
		if i < len(stations)-1 {
			fmt.Print(", ")
		}
	}
	fmt.Println("}")
}

Results#

The concurrent version, unexpectedly, resulted in almost a 3x decrease in performance.

go run main.go  84.15s user 101.34s system 342% cpu 54.225 total

Where did we go wrong? This is a classic case where the overhead of concurrency mechanisms outweighs their benefits. In our current implementation, each line is sent to the channel individually, which is likely less efficient than batching lines for processing. This means that for a file with a large number of lines, there will be an equally large number of channel send operations. Each channel operation involves locking and unlocking, which can be costly, especially in a high-frequency context.

Iteration 2: Batch processing of lines#

In this version we are Batching the lines before sending to the worker which will significantly reduce the overhead of channel communication.

1. Batch Processing: Each batch contains batchSize lines. This reduces the frequency of channel operations (both sending and receiving), as well as the overhead associated with these operations.

2. Efficient Worker Utilization: With batch processing, each worker goroutine spends more time processing data and less time interacting with channels. This reduces the overhead of context switching and synchronization, making the processing more efficient.

const (
	batchSize        = 1000000 // Number of lines per batch
)

// ...
		scanner := bufio.NewScanner(file)
		var batch []string
		for scanner.Scan() {
			batch = append(batch, scanner.Text())
			if len(batch) >= batchSize {
				batchesChan <- batch
				batch = nil // Start a new batch
			}
		}
		// Send any remaining lines in the last batch
		if len(batch) > 0 {
			batchesChan <- batch
		}
		close(batchesChan)

// ...
func worker(batchesChan <-chan []string, resultsChan chan<- map[string]Stats, wg *sync.WaitGroup) {
	defer wg.Done()

	stationStats := make(map[string]Stats)
	for batch := range batchesChan {
		for _, line := range batch {
            // Process the line ...
        }
	}

	resultsChan <- stationStats
}
// ...

Results#

The improvement from iteration 2 to iteration 3 is quite remarkable, thanks to efficiently batching the lines together and reducing the number of channel ops.

go run main.go  30.02s user 0.67s system 476% cpu 6.442 total

So far, we’ve reduced the time to about 6.5s which is a great start and improvement of our baseline version of 19s. However, we’re making quite a few extra memory allocations and the focus of next iteration should be to reduce that.

Iteration 3: Reducing memory allocations#

• A batch slice is pre-allocated with a capacity of batchSize and reused for each batch of lines.
• After sending a batch to the channel, the slice is reset to zero length (batch = batch[:0]), but the underlying array is retained and reused.

// Read the file and send batches of lines to the workers
	go func() {
		file, err := os.Open(measurementsFile)
		if err != nil {
			panic(err)
		}
		defer file.Close()

		scanner := bufio.NewScanner(file)
		batch := make([]string, 0, batchSize) // Pre-allocate with capacity

		for scanner.Scan() {
			line := scanner.Text()

			// Reuse the batch slice by appending to it until it reaches the batch size
			batch = append(batch, line)

			if len(batch) >= batchSize {
				batchesChan <- batch
				batch = batch[:0] // Reset the slice without allocating new memory
			}
		}
		// Send any remaining lines in the last batch
		if len(batch) > 0 {
			batchesChan <- batch
		}
		close(batchesChan)
	}()

Results#

Down to 5.3s!

go run main.go  25.43s user 0.53s system 485% cpu 5.346 total

Iteration 3 (cont): Further reducing memory allocations#

• Avoiding strings.Split: Instead of using strings.Split, which allocates a new slice for each line, we can use strings.Index to find the delimiter and manually slice the string. strings.Split typically creates a new slice for each split part, leading to more memory usage and subsequent GC overhead.

for batch := range batchesChan {
		for _, line := range batch {
			delimiterIndex := strings.Index(line, ";")
			if delimiterIndex == -1 {
				continue // Delimiter not found, skip this line
			}

			station := line[:delimiterIndex]

			tempStr := line[delimiterIndex+1:]
			temp, err := strconv.ParseFloat(tempStr, 64)
			if err != nil {
				continue // Invalid temperature value, skip this line
			}

			stats := stationStats[station]
			stats.Count++
			stats.Min = min(stats.Min, temp)
			stats.Max = max(stats.Max, temp)
			stats.Mean += (temp - stats.Mean) / float64(stats.Count)
			stationStats[station] = stats
		}
	}

Results#

The time has further decreased from 5.3s to 4.8s with these changes.

go run main.go  15.69s user 0.44s system 332% cpu 4.853 total

Iteration 4: Read file in chunks#

In this version, the file is read in chunks, and each chunk is processed to ensure it contains complete lines. The processChunk function is used to separate valid data from leftover data in each chunk. Chunk size can be controlled with command line args as well.

func main() {
	// ....
	const chunkSize = 256 * 1024 // 256 KB
	buf := make([]byte, chunkSize)
	leftover := make([]byte, 0, chunkSize)

	go func() {
		for {
			bytesRead, err := file.Read(buf)
			if bytesRead > 0 {
				// Copy the chunk to a new slice, because the
				// buffer will be reused in the next iteration.
				chunk := make([]byte, bytesRead)
				copy(chunk, buf[:bytesRead])
				// Process the chunk. The returned leftover will be processed in the next iteration.
				validChunk, newLeftover := processChunk(chunk, leftover)
				leftover = newLeftover
				// Send the valid chunk to the processing goroutine.
				if len(validChunk) > 0 {
					wg.Add(1)
					go processChunkData(validChunk, resultsChan, &wg)
				}
			}
			if err != nil {
				break
			}
		}
		wg.Wait()
		close(resultsChan)
	}()
	// ...
}


func processChunk(chunk, leftover []byte) (validChunk, newLeftover []byte) {
	firstNewline := -1
	lastNewline := -1
	// Find the first and last newline in the chunk.
	for i, b := range chunk {
		if b == '\n' {
			if firstNewline == -1 {
				firstNewline = i
			}
			lastNewline = i
		}
	}
	if firstNewline != -1 {
		validChunk = append(leftover, chunk[:lastNewline+1]...)
		newLeftover = make([]byte, len(chunk[lastNewline+1:]))
		copy(newLeftover, chunk[lastNewline+1:])
	} else {
		newLeftover = append(leftover, chunk...)
	}
	return validChunk, newLeftover
}

func processChunkData(chunk []byte, resultsChan chan<- map[string]Stats, wg *sync.WaitGroup) {
	defer wg.Done()

	stationStats := make(map[string]Stats)
	scanner := bufio.NewScanner(strings.NewReader(string(chunk)))

	for scanner.Scan() {
		line := scanner.Text()

		// Find the index of the delimiter
		delimiterIndex := strings.Index(line, ";")
		if delimiterIndex == -1 {
			continue // Delimiter not found, skip this line
		}

		// Extract the station name and temperature string
		station := line[:delimiterIndex]
		tempStr := line[delimiterIndex+1:]

		// Convert the temperature string to a float
		temp, err := strconv.ParseFloat(tempStr, 64)
		if err != nil {
			continue // Invalid temperature value, skip this line
		}

		// Update the statistics for the station
		stats, exists := stationStats[station]
		if !exists {
			stats = Stats{Min: temp, Max: temp}
		}
		stats.Count++
		stats.Min = min(stats.Min, temp)
		stats.Max = max(stats.Max, temp)
		stats.Mean += (temp - stats.Mean) / float64(stats.Count)
		stationStats[station] = stats
	}

	// Send the computed stats to resultsChan
	resultsChan <- stationStats
}

In addition to this, I moved the aggregateStats to a separate Goroutine as well:

	aggWg.Add(1)
	finalResults := make(map[string]Stats)

	// Start a separate goroutine for aggregation
	go func() {
		defer aggWg.Done()
		for workerResult := range resultsChan {
			for station, stats := range workerResult {
				finalStats, exists := finalResults[station]
				if !exists {
					finalResults[station] = stats
					continue
				}
				finalStats.Min = min(finalStats.Min, stats.Min)
				finalStats.Max = max(finalStats.Max, stats.Max)
				totalCount := finalStats.Count + stats.Count
				finalStats.Mean = (finalStats.Mean*float64(finalStats.Count) + stats.Mean*float64(stats.Count)) / float64(totalCount)
				finalStats.Count = totalCount
				finalResults[station] = finalStats
			}
		}
	}()

Results#

We’re down from 4.8s to just 2.1s to read/parse/process 100mn lines!

./bin/1brc.bin --file=input.txt --chunksize=1048576  17.58s user 0.77s system 837% cpu 2.190 total

Summary#

1. Basic File Reading and Parsing (Baseline):

   • Time: 19s (baseline).
   • Key Change: Sequentially reading and processing each line.
   • Speedup: N/A (baseline).

2. Producer-Consumer Pattern:

   • Time: 54.225s.
   • Key Change: Implemented concurrent line processing with producer-consumer pattern.
   • Speedup: -185% (slower than baseline).

3. Batch Processing of Lines:

   • Time: 6.442s.
   • Key Change: Batched lines before processing, reducing channel communication.
   • Speedup: +66% (compared to baseline).

4. Reducing Memory Allocations - Iteration 1:

   • Time: 5.346s.
   • Key Change: Reused batch slices and reduced memory allocations.
   • Speedup: +72% (compared to baseline).

5. Reducing Memory Allocations - Iteration 2 (Avoiding strings.Split):

   • Time: 4.853s.
   • Key Change: Replaced strings.Split with manual slicing for efficiency.
   • Speedup: +75% (compared to baseline).

6. Read File in Chunks:

   • Time: 2.190s.
   • Key Change: Processed file in chunks and optimized aggregation.
   • Speedup: +87% (compared to baseline).

Final Run#

I’m quite satisfied with the final version for now. We can now proceed to test it with 1 billion lines. However it’s evidently CPU-bound, as we spawn N workers for N CPUs.

I experimented with different chunk sizes, and here are the results from each run:

Chunk Size	Time
512.00 KB	23.756s
1.00 MB	21.798s
16.00 MB	20.693s
32.00 MB	19.501s

Tweaking the chunk size doesn’t significantly impact performance, as processing larger chunks takes longer.

TL;DR: On an average and with multiple runs it takes approx 20s with the final iteration for 1bn lines.

Checkout the full code on my GitHub.

Potential Improvements#

This project was not only fun but also a great opportunity to revisit and refine many Go concepts. There are several ideas to contemplate for further improving this version’s timings:

• I haven’t yet considered using mmap, but I believe it could substantially speed things up.
• To delve even deeper, custom line parsing functions, especially for converting string to float64, could offer improvements.
• Employing custom hashing functions (perhaps FnV) might aid in faster map lookups.

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/sad-servers/ https://mrkaran.dev/posts/sad-servers/ Mon, 06 Nov 2023 00:00:00 GMT Introduction to SadServers#

Recently, I stumbled upon sadservers, a platform described as “Like LeetCode for Linux”. The premise is: you are given access to a full remote Linux server with a pre-configured problem. Your mission is to diagnose and fix the issues in a fixed time window.

With the goal of documenting my journey through these challenges and sharing the knowledge gained, I decided to not only tackle these puzzles but also to record my solutions in a video format. The format is twofold in its purpose: it allows me to reflect on my problem-solving approach and provides a resource for others who may encounter similar problems, whether in real-world scenarios or in preparation for an SRE/DevOps interview.

The Learning Curve#

Each server presented a different issue, from misconfigured network settings to services failing to start, from permission issues to resource overutilization. One server, for instance, had a failing database service because of a disk full partition. The cause? Stale backup files. Another had a web server throwing errors because of incorrect file permissions.

Recording the Solutions#

The video recordings start with an introduction to the problem and my initial thoughts. Viewers can see my screen as I work through the issue, making the troubleshooting process transparent and educational. The commentary explains my thought process, the tools/CLI utilities used, and the solutions applied.

Part 1#

Part 2#

Part 3#

Conclusion#

For those looking to enhance their Linux troubleshooting skills, sadservers.com is a gold mine. It’s an excellent preparation ground for anyone aiming to step into the SRE/DevOps field or wanting to keep their skills sharp.

As I continue to record and share these troubleshooting escapades, I invite you to subscribe, comment with your insights, or even suggest what types of challenges you’d like to see addressed next.

]]> Karan Sharma https://aryak.me/blog/05-rupee-sign-on-hyprland.html https://aryak.me/blog/05-rupee-sign-on-hyprland.html Fri, 03 Nov 2023 12:39:45 GMT Enabling the rupee sign (or the euro sign for that matter) on Hyprland is pretty simple, but not well documented from my “research”.

To begin with, you need to use the altgr-intl layout in order to be able to use it in the first place. This also gives you access to many other characters as well.

To do this, add kb_variant = altgr-intl in the input section of your hyprland.conf.

Past this, the configuration is pretty simple, you just have to add the required options to kb_options.

You can get a list of these with localectl list-x11-keymap-options.

In my case, I needed rupeesign:4

At the end, this is how the hyprland.conf’s input section looks:

input {
    kb_layout = us
    kb_options = rupeesign:4, caps:backspace
    kb_variant = altgr-intl
    [...mouse stuff...]
}

]]> arya@projectsegfau.lt (Arya Kiran) 2023/11/03/5 https://ravidwivedi.in/posts/software-freedom-day-at-sflc.in/ https://ravidwivedi.in/posts/software-freedom-day-at-sflc.in/ Sun, 22 Oct 2023 21:55:34 GMT Software Freedom Law Center, India, also known as sflc.in, organized an event to celebrate the Software Freedom Day on 30th September 2023. I, Sahil, Contrapunctus and Suresh joined. The venue was at the SFLC India office in Delhi. The sflc.in office was on the second floor of what looked like someone’s apartment:). I also met Chirag, Orendra, Surbhi and others.

My plan was to have a stall on LibreOffice and Prav app to raise awareness about these projects. I didn’t have QR code for downloading prav app printed already, so I asked the people at sflc.in if they can get it printed for me. They were very kind and helped me in getting a color printout for me. So, I got a stall in their main room. Surbhi was having an Inkscape stall next to mine and gave me company. People came and asked about the prav project and then I realized I was still too tired to explain the idea behind the prav project and about LibreOffice (after a long Kerala trip). We got a few prav app installs during the event, which is cool.

Sahil had a Debian stall, and contrapunctus had an OpenStreetMap stall. After about an hour, Revolution OS was screened for all of us to watch, along with popcorn. The documentary gave an overview of history of Free Software Movement. The office had a kitchen where fresh chai was being made and served to us. The organizers ordered a lot of good snacks for us.

I came out of the movie hall to take more tea and snacks from the front desk. I saw a beautiful painting was hanging at the wall opposite to the front desk and Tejaswini (from sflc.in) revealed that she had made it. The tea was really good as it was freshly made in the kitchen.

After the movie, we played a game of pictionary. We were divided into two teams. The game goes as follows: A person from a team is selected and given a term related to freedom respecting software written on a piece of paper, but concealed from other participants. Then that person draws something on the board (no logo, no alphabets) without speaking. If the person’s team correctly guesses the term, the team gets one step ahead on the leaderboard. The team that reaches the finish line wins.

I recall some fun Pictionary rounds. For example, the one in the picture below seemed far from the word “Wireguard,” but someone from the team still guessed that word. Our team won in the end \o/.

Then, we posed for a group picture. At the end, SFLC.in had a delicious cake in store for us. They also had some merchandise available, such as handbags, T-shirts, etc., which we could take if we made a donation to SFLC.in. I “bought” a handbag with “Ban Plastic, not Internet” written on it in exchange for a donation. I hope that gives people around me a powerful message :)

Overall, sflc.in hosted a fantastic event!

]]> Ravi Dwivedi https://www.evalapply.org/posts/mycelium-clj/index.html https://www.evalapply.org/posts/mycelium-clj/index.html Thu, 19 Oct 2023 00:00:00 GMT Aditya Athalye riff clojure systems architecture recurse_center https://www.prashanthudupa.com/an-approach-to-getting-fit/ https://www.prashanthudupa.com/an-approach-to-getting-fit/ Tue, 17 Oct 2023 08:11:52 GMT Prashanth Udupa Fitness Philosophy https://ravidwivedi.in/posts/kochi-wayanad-trip-aug-sep-2023/ https://ravidwivedi.in/posts/kochi-wayanad-trip-aug-sep-2023/ Sat, 14 Oct 2023 12:18:54 GMT A trip full of hitchhiking, beautiful places and welcoming locals.

Day 1: Arrival in Kochi

Kochi is a city in the state of Kerala, India. This year’s DebConf was to be held in Kochi from 3rd September to 17th of September, which I was planning to attend. My friend Suresh, who was planning to join, told me that 29th August 2023 will be Onam, a major festival of the state of Kerala. So, we planned a Kerala trip before the DebConf. We booked early morning flights for Kochi from Delhi and reached Kochi on 28th August.

We had booked a hostel named Zostel in Ernakulam. During check-in, they asked me to fill a form which required signing in using a Google account. I told them I don’t have a Google account and I don’t want to create one either. The people at the front desk seemed receptive, so I went ahead with telling them the problems of such a sign-in being mandatory for check-in. Anyways, they only took a photo of my passport and let me check-in without a Google account.

We stayed in a ten room dormitory, which allowed travellers of any gender. The dormitory room was air-conditioned, spacious, clean and beds were also comfortable. There were two bathrooms in the dormitory and they were clean. Plus, there was a separate dormitory room in the hostel exclusive for females. I noticed that that Zostel was not added in the OpenStreetMap and so, I added it :) . The hostel had a small canteen for tea and snacks, a common sitting area outside the dormitories, which had beds too. There was a separate silent room, suitable for people who want to work.

We had lunch at a nearby restaurant and it was hard to find anything vegetarian for me. I bought some freshly made banana chips from the street and they were tasty. As far as I remember, I had a big glass of pineapple juice for lunch. Then I went to the Broadway market and bought some cardamom and cinnamon for home. I also went to a nearby supermarket and bought Matta brown rice for home. Then, I looked for a courier shop to send the things home but all of them were closed due to Onam festival. After returning to the Zostel, I overslept till 9 PM and in the meanwhile, Suresh planned with Saidut and Shwetank (who met us during our stay in Zostel) to go to a place in Fort Kochi for dinner. I suspected I will be disappointed by lack of vegetarian options as they were planning to have fish. I already had a restaurant in mind - Brindhavan restaurant (suggested by Anupa), which was a pure vegetarian restaurant.

To reach there, I got off at Palarivattom metro station and started looking for an auto-rickshaw to get to the restaurant. I didn’t get any for more than 5 minutes. Since that restaurant was not added to the OpenStreetMap, I didn’t even know how far that was and which direction to go to. Then, I saw a Zomato delivery person on a motorcycle and asked him where the restaurant was. It was already 10 PM and the restaurant closes at 10:30. So, I asked him whether he can drop me off. He agreed and dropped me off at that restaurant. It was 4-5 km from that metro station. I tipped him and expressed my gratefulness for the help. He refused to take the tip, but I insisted and he accepted.

I entered the restaurant and it was coming to a close, so many items were not available. I ordered some Kadhai Paneer (only item left) with naan. It tasted fine. Since the next day was Thiruvonam, I asked the restaurant about the Sadya thali menu and prices for the next day. I planned to eat Sadya thali at that restaurant, but my plans got changed later.

Day 2: Onam celebrations

Next day, on 29th of August 2023, we had plan to leave for Wayanad. Wayanad is a hill station in Kerala and a famous tourist spot. Praveen suggested to visit Munnar as it is far closer to Kochi than Wayanad (80 km vs 250 km). But I had already visited Munnar in my previous trips, so we chose Wayanad. We had a train late night from Ernakulam Junction (at 23:30 hours) to Kozhikode, which is the nearest railway station from Wayanad. So, we checked out in the morning as we had plans to roam around in Kochi before taking the train.

Zostel was celebrating Onam on that day. To opt-in, we had to pay 400 rupees, which included a Sadya Thali and a Mundu. Me and Suresh paid the amount and opted in for the celebrations. Sadya thali had Rice, Sambhar, Rasam, Avial, Banana Chips, Pineapple Pachadi, Pappadam, many types of pickels and chutneys, Pal Ada Payasam and Coconut jaggery Pasam. And, there was water too :). Those payasams were really great and I had one more round of them. Later, I had a lot of variety of payasams during the DebConf.

In the evening, we hung out in the common room and put our luggage there. We played UNO and had conversations with other travellers in the hostel. I had a fun time there and I still think it is one of the best hostel experiences I had. We made good friends with Saiduth (Telangana) and Shwetank (Uttarakhand). They were already aware about the software like debian, and we had some detailed conversations about the Free Software movement. I remember explaining the difference between the terms “Open Source” and “Free Software”. I also told them about the Streetcomplete app, a beginner friendly app to edit OpenStreetMap. We had dinner at a place nearby (named Palaraam), but again, the vegetarian options were very limited! After dinner, we came back to the Zostel and me and Suresh left for Ernakulam Junction to catch our train Maveli Express (16604).

Day 3: Going to Wayanad

Maveli Express was scheduled to reach Kozhikode at 03:25 (morning). I had set alarms from 03:00 to 03:30, with the gap of 10 minutes. Every time I woke up, I turned off the alarm. Then I woke up and saw train reaching the Kozhikode station and woke up Suresh for deboarding. But then I noticed that the train is actually leaving the station, not arriving! This means we missed our stop. Now we looked at the next stops and whether we can deboard there. I was very sleepy and wanted to take a retiring room at some station before continuing our journey to Wayanad. The next stop was Quilandi and we checked online that it didn’t have a retiring room. So, we skipped this stop. We got off at the next stop named Vadakara and found out no retiring room was available. So, we asked about information regarding bus for Wayanad and they said that there is a bus to Wayanad around 07:00 hours from bus station which was a few kilometres from the railway station.

We took a bus for Kalpetta (in Wayanad) at around 07:00. The destination of the buses were written in Malayalam, which we could not read. Once again, the locals helped us to get on to the bus to Kalpetta. Vadakara is not a big city and it can be hard to find people who know good Hindi or English, unlike Kochi. Despite language issues, I had no problem there in navigation, thanks to locals. I mostly spent time sleeping during the bus journey.

A few hours later, the bus dropped us at Kalpetta. We had a booking at a hostel in Rippon village. It was 16 km from Kalpetta. On the way, we were treated with beautiful views of nature, which was present everywhere in Wayanad. The place was covered with tea gardens and our eyes were treated with beautiful scenery at every corner.

Rippon village was a very quiet place and I liked the calm atmosphere. This place is blessed by nature and has stunning scenery. I found English was more common than Hindi in Wayanad. Locals were very nice and helpful, even if they didn’t know my language.

After catching some sleep at the hostel, I went out in the afternoon. I hitchhiked to reach the main road from the hostel. I bought more spices from a nearby shop and realized that I should have waited for my visit to Wayanad to buy cardamom, which I already bought from Kochi. Then, I was looking for post office to send spices home. The people at the spices shop told me that the nearby Rippon post office was closed by that time, but the post office at Meppadi was open, which was 5 km from there.

I went to Meppadi and saw the post office closes at 15:00, but I reached five minutes late. My packing was not very good and they asked me to pack it tighter. There was a shop near the post office and the people there gave me a cardboard and tapes, and helped pack my stuff for the post. By the time I went to the post office again, it was 15:30. But they accepted my parcel for post.

Day 4: Kanthanpara Falls, Zostel Wayanad and Karapuzha Dam

Kanthanpara waterfalls were 2 km from the hostel. I hitchhiked to the place from the hostel on a scooty. Entry ticket was worth Rs 40. There were good views inside and nothing much to see except the waterfalls.

We had a booking at Zostel Wayanad for this day and so we shifted there. Again, as with their Ernakulam branch, they asked me to fill a form which required signing in using Google, but when I said I don’t have a Google account they checked me in without that. There were tea gardens inside the Zostel boundaries and the property was beautiful.

Later in the evening, I went to Karapuzha Dam. I witnessed a beautiful sunset during the journey. Karapuzha dam had many activites, like ziplining, and was nice to roam around.

Chembra Peak is near to the Zostel Wayanad. So, I was planning to trek to the heart shaped lake. It was suggested by Praveen and looking online, this trek seemed worth doing. There was an issue however. The charges for trek were Rs 1770 for upto five people. So, if I go alone I will have to spend Rs 1770 for the trek. If I go with another person, we split Rs 1770 into two, and so on. The optimal way to do it is to go in a group of five (you included :D). I asked front desk at Zostel if they can connect me with people going to Chembra peak the next day, and they told me about a group of four people planning to go to Chembra peak the next day. I got lucky! All four of them were from Kerala and worked in Qatar.

Day 5: Chembra peak trek

The date was 1st September 2023. I woke up early (05:30 in the morning) for the Chembra peak trek. I had bought hiking shoes especially for trekking, which turned out to be a very good idea. The ticket counter opens at 07:00. The group of four with which I planned to trek met me around 06:00 in the Zostel. We went to the ticket counter around 06:30. We had breakfast at shops selling Maggi noodles and bread omlette near the ticket counter.

It was a hot day and the trek was difficult for an inexperienced person like me. The scenery was green and beautiful throughout.

The day was scorching hot, and the trek proved to be challenging for someone like me, who was relatively inexperienced. However, the lush green scenery surrounding us was incredibly beautiful throughout the journey.

On the way back from the trek, I stumbled upon a shop selling bamboo rice, which I purchased with the intention of making bamboo rice payasam at home (I even have some coconut milk from Kerala to use ;)). We arrived back at Zostel in the afternoon. I experienced muscle soreness after the trek, and it has yet to completely subside. Later that night, we boarded a bus from Kalpetta to Kozhikode to begin our journey back to Kochi.

Day 6: Return to Kochi

At midnight on the 2nd of September, we arrived at Kozhikode bus stand. Afterward, we wandered around in search of something to eat, but unfortunately, I couldn’t find any vegetarian options. Not a surprising turn of events, considering Kozhikode is especially famous for its non-vegetarian dishes. We then headed to Kozhikode railway station to inquire about retiring rooms, but we were out of luck. We waited at the station and caught the next train to Kochi at 03:30, arriving at Ernakulam Junction at 07:30, half an hour before the scheduled arrival time of the train. From there, we made our way to Zostel Fort Kochi, where we spent the night before checking out the next morning.

Day 7: Roaming around in Fort Kochi

On the 3rd of September, we explored Fort Kochi, visiting popular landmarks such as St. Francis Church, Dutch Palace, Jew Town, and the Pardesi Synagogue. During our visit, I also had the opportunity to tour some homestays, where the owners graciously showed me around their properties, despite my clear indication that I was not seeking accommodation. In the evening, we made our way to Kakkanad to participate in DebConf.

For more details on my experiences at DebConf23, you can continue reading in my DebConf23 blog post.

]]> Ravi Dwivedi https://www.learningfromdata.zingg.ai/p/zingg-turns-two https://www.learningfromdata.zingg.ai/p/zingg-turns-two Sat, 30 Sep 2023 17:09:19 GMT Two years is a journey, yet it seems like yesterday when I open sourced Zingg. In the interim, we have done fulfilling work and received a lot of love. Curious what’s up at our end? Read on!

First thing first.

We now have paying customers for our enterprise products! There is nothing more exhilarating than seeing our life’s work come alive. And to be paid so that we can continue to build and move in the direction we aspire to. In the startup world, we hold a lot of discussions around product market fit, product founder fit, building things people want, user research, and other associated activities. Product-led startups like ours get the first set of customers through the urgent need of a visionary early adopter. The visionary early adopter is the person in a team who has a pressing need along with the foresight to solve the problem and the confidence to trust a new team and an innovative product. And who can influence and move budgets to solve that need. I am very glad to have this opportunity to work closely with this first set of customers who are helping us shape the product. Not just the features, but also the pricing.

Thank you for supporting us so that we can help you!

We are building three flavors of Zingg Enterprise. Zingg Snowflake, Zingg Databricks and Zingg Spark. All of them with the same roots in Zingg open source. Each carries forward Zingg OSS and gives the required ammunition to build and use the persistent and evolving enterprise identity graph. Zingg Enterprise creates global unique identifiers for entities, explains model behaviour and maintains and enhances the identity graph through resolution of new and updated records against previously matched records. Of course there is the improved accuracy and scalability :-)

On top of this, we have added platform-specific features. The Snowflake version runs inside Snowflake without any other infrastructure or software dependence. The Databricks flavour adds Unity Catalog support and a lot of other Databricks goodies. The Spark one has the enterprise features and runs on all other Spark environments like EMR.

On the open source side, we just hit 1000 downloads per month on PyPi! The rate of downloads has doubled over the past few months, and we now have a Slack group touching 450 people, from varied organizations like LinkedIn, Maersk, Newfront, Samba TV, Canadian Football League, Heb and John Deere to name a few. We have federal agencies like Pandemic Response Centre, and established startups like Provenance.org, Redica Systems, Ninjacart and Cherre. We have a good number of nonprofits and data consultancies as well as friends from Databricks, Snowflake and Exasol. The strong code contributions from these partner companies have helped us to support their users more deeply.

We are clearly onto something :-)

I am truly amazed by the depth of use cases we are seeing across big and small organizations in varying domains. Zingg is resolving customer identity for marketing and personalization use cases in retail. It is resolving patient identity for comprehensive health, and healthcare provider identity to understand drug recommendation patterns. And the identity of an insurance policyholder, for compliance and risk, and also from the point of view of a reinsurer. Identity resolution is much broader than we thought initially. It is the identity of members of religious establishments, of donors or recipients. It is the identity of the fans of a football club to send out the right information at the right time for a game of their liking. B2B accounts, which buy from multiple subsidiaries in different geographies, need identity resolution as well. So do products on an e-commerce website for competitive analysis on other sites.

Identity is anything and everything a business cares about.

We are happy that we built Zingg so that users can work on their schema and train on their data without specialized ML skills. All in a privacy centric way. This has allowed us to see Zingg in action in the different use cases we are discovering every passing day.

To each of you who has used Zingg, mentioned us to friends and coworkers or checked us out - thank you for your time and we hope we have removed some data troubles from your life.

I also wanted to thank our investors and advisors, who backed Zingg when no one had heard of us, and who have been generous with their time and support.

These two years have been a tremendous learning experience, and our plans are only getting bigger. There is a lot of AI yet to be applied to enterprise data, and we will love to create more value for our users and customers. For that, we need to continue to work closely with them, ship more and more often, keep the momentum going and work twice as hard.

We are excited for the path ahead. Wish us luck!

]]> Sonal Goyal https://ravidwivedi.in/posts/fixing-audio-and-keymap-in-chromebook-running-debian/ https://ravidwivedi.in/posts/fixing-audio-and-keymap-in-chromebook-running-debian/ Tue, 26 Sep 2023 07:37:22 GMT I recently bought an HP Chromebook from Abhas who had already flashed coreboot in it. I ran a fresh installation of Debian 12 (Bookworm) on it with KDE Plasma.

Right after installation, the Wi-Fi and bluetooth were working, but I was facing two issues:

• Playing a music file or any audio file does not give any audio.

• Keyboard buttons like the ones for brightness and audio adjustment were not working (alphabet keys were working).

Fixing audio

I ran the script mentioned here and that fixed the audio.

The instructions from that link are:

git clone https://github.com/WeirdTreeThing/chromebook-linux-audio
cd chromebook-linux-audio
./setup-audio

Fixing keyboard

To fix the keyboard, go to KDE Settings and in the Hardware section, under the “Keyboard model” box select the option Google | Chromebook as depicted in the screenshot below.

I hope this post fixed the issues for you. Meet you in the next post.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/debconf23/ https://ravidwivedi.in/posts/debconf23/ Fri, 22 Sep 2023 18:19:38 GMT

Introduction

DebConf23, the 24th annual Debian Conference, was held in India in the city of Kochi, Kerala from the 3rd to the 17th of September, 2023. Ever since I got to know about it (which was more than an year ago), I was excited to attend DebConf in my home country. This was my second DebConf, as I attended one last year in Kosovo. I was very happy that I didn’t need to apply for a visa to attend. I got full bursary to attend the event (thanks a lot to Debian for that!) which is always helpful in covering the expenses, especially if the venue is a five star hotel :)

For the conference, I submitted two talks. One was suggested by Sahil on Debian packaging for beginners, while the other was suggested by Praveen who opined that a talk covering broader topics about “freedom” in self-hosting services will be better, when I started discussing about submitting a talk about prav app project. So I submitted one on Debian packaging for beginners and the other on ideas on sustainable solutions for self-hosting.

My friend Suresh - who is enthusiastic about Debian and free software - wanted to attend the DebConf as well. When the registration started, I reminded him about applying. We landed in Kochi on the 28th of August 2023 during the festival of Onam. We celebrated Onam in Kochi, had a trip to Wayanad, and returned to Kochi. On the evening of the 3rd of September, we reached the venue - Four Points Hotel by Sheraton, at Infopark Kochi, Ernakulam, Kerala, India.

Hotel overview

The hotel had 14 floors, and featured a swimming pool and gym (these were included in our package). The hotel gave us elevator access for only our floor, along with public spaces like the reception, gym, swimming pool, and dining areas. The temperature inside the hotel was pretty cold and I had to buy a jacket to survive. Perhaps the hotel was in cahoots with winterwear companies? :)

Meals

On the first day, Suresh and I had dinner at the eatery on the third floor. At the entrance, a member of the hotel staff asked us about how many people we wanted a table for. I told her that it’s just the two of us at the moment, but (as we are attending a conference) we might be joined by others. Regardless, they gave us a table for just two. Within a few minutes, we were joined by Alper from Turkey and urbec from Germany. So we shifted to a larger table…but then we were joined by even more people, so we were busy adding more chairs to our table. urbec had already been in Kerala for the past 5-6 days and was, on one hand, very happy already with the quality and taste of bananas in Kerala…and on the other, rather afraid of the spicy food :)

Two days later, the lunch and dinner were shifted to the All Spice Restaurant on the 14th floor, but the breakfast was still served at the eatery. Since the eatery (on the 3rd floor) had greater variety of food than the other venue, this move made breakfast the best meal for me and many others. Many attendees from outside India were not accustomed to the “spicy” food. It is difficult for locals to help them, because what we consider mild can be spicy for others. It is not easy to satisfy everyone at the dining table, but I think the organizing team did a very good job in the food department. (That said, it didn’t matter for me after a point, and you will know why.) The pappadam were really good, and I liked the rice labelled “Kerala rice”. I actually brought that exact rice and pappadam home during my last trip to Kochi and everyone at my home liked it too (thanks to Abhijit PA). I also wished to eat all types of payasams from Kerala and this really happened (thanks to Sruthi who designed the menu). Every meal had a different variety of payasam and it was awesome, although I didn’t like some of them, mostly because they were very sweet. Meals were later shifted to the ground floor (taking away the best breakfast option which was the eatery).

The excellent Swag Bag

The DebConf registration desk was at the second floor. We were given a very nice swag bag. They were available in multiple colors - grey, green, blue, red - and included an umbrella, a steel mug, a multiboot USB drive by Mostly Harmless, a thermal flask, a mug by Canonical, a paper coaster, and stickers. It rained almost every day in Kochi during our stay, so handing out an umbrella to every attendee was a good idea.

A gift for Nattie

During breakfast one day, Nattie (Belgium) expressed the desire to buy a coffee filter. The next time I went to the market, I bought a coffee filter for her as a gift. She seemed happy with the gift and was flattered to receive a gift from a young man :)

Being a mentor

There were many newbies who were eager to learn and contribute to Debian. So, I mentored whoever came to me and was interested in learning. I conducted a packaging workshop in the bootcamp, but could only cover how to set up the Debian Unstable environment, and had to leave out how to package (but I covered that in my talk). Carlos (Brazil) gave a keysigning session in the bootcamp. Praveen was also mentoring in the bootcamp. I helped people understand why we sign GPG keys and how to sign them. I planned to take a workshop on it but cancelled it later.

My talk

My Debian packaging talk was on the 10th of September, 2023. I had not prepared slides for my Debian packaging talk in advance - I thought that I could do it during the trip, but I didn’t get the time…so I prepared them on the day before the talk. Since it was mostly a tutorial, the slides did not need much preparation. My thanks to Suresh, who helped me with the slides and made it possible to complete them in such a short time frame.

My talk was well-received by the audience, going by their comments. I am glad that I could give an interesting presentation.

Visiting a saree shop

After my talk, Suresh, Alper, and I went with Anisa and Kristi - who are both from Albania, and have a never-ending fascination for Indian culture :) - to buy them sarees. We took autos to Kakkanad market and found a shop with a great variety of sarees. I was slightly familiar with the area around the hotel, as I had been there for a week. Indian women usually don’t try on sarees while buying - they just select the design. But Anisa wanted to put one on and take a few photos as well. The shop staff did not have a trial saree for this purpose, so they took a saree from a mannequin. It took about an hour for the lady at the shop to help Anisa put on that saree…but you could tell that she was in heaven wearing that saree, and she bought it immediately :) Alper also bought a saree to take back to Turkey for his mother. Me and Suresh wanted to buy a kurta which would go well with the mundu we already had, but we could not find anything to our liking.

Cheese and Wine Party

On the 11th of September we had the Cheese and Wine Party, a tradition of every DebConf. I brought Kaju Samosa and Nankhatai from home. Many attendees expressed their appreciation for the samosas. During the party, I was with Abhas and had a lot of fun. Abhas brought packets of paan and served them at the Cheese and Wine Party. We discussed interesting things and ate burgers. But due to the restrictive alcohol laws in the state, it was less fun compared to the previous DebConfs - you could only drink alcohol served by the hotel in public places. If you bought your own alcohol, you could only drink in private places (such as in your room, or a friend’s room), but not in public places.

Party at my room

Last year, Joenio (Brazilian) brought pastis from France which I liked. He brought the same alocholic drink this year too. So I invited him to my room after the Cheese and Wine party to have pastis. My idea was to have them with my roommate Suresh and Joenio. But then we permitted Joenio to bring as many people as he wanted…and he ended up bringing some ten people. Suddenly, the room was crowded. I was having good time at the party, serving them the snacks given to me by Abhas. The news of an alcohol party at my room spread like wildfire. Soon there were so many people that the AC became ineffective and I found myself sweating.

I left the room and roamed around in the hotel for some fresh air. I came back after about 1.5 hours - for most part, I was sitting at the ground floor with TK Saurabh. And then I met Abraham near the gym (which was my last meeting with him). I came back to my room at around 2:30 AM. Nobody seemed to have realized that I was gone. They were thanking me for hosting such a good party. A lot of people left at that point and the remaining people were playing songs and dancing (everyone was dancing all along!). I had no energy left to dance and to join them. They left around 03:00 AM. But I am glad that people enjoyed partying in my room.

Sadhya Thali

On the 12th of September, we had a sadhya thali for lunch. It is a vegetarian thali served on a banana leaf on the eve of Thiruvonam. It wasn’t Thiruvonam on this day, but we got a special and filling lunch. The rasam and payasam were especially yummy.

Day trip

On the 13th of September, we had a daytrip. I chose the daytrip houseboat in Allepey. Suresh chose the same, and we registered for it as soon as it was open. This was the most sought-after daytrip by the DebConf attendees - around 80 people registered for it.

Our bus was set to leave at 9 AM on the 13th of September. Me and Suresh woke up at 8:40 and hurried to get to the bus in time. It took two hours to reach the venue where we get the houseboat.

The houseboat experience was good. The trip featured some good scenery. I got to experience the renowned Kerala backwaters. We were served food on the boat. We also stopped at a place and had coconut water. By evening, we came back to the place where we had boarded the boat.

A good friend lost

When we came back from the daytrip, we received news that Abhraham Raji was involved in a fatal accident during a kayaking trip.

Abraham Raji was a very good friend of mine. In my Albania-Kosovo-Dubai trip last year, he was my roommate at our Tirana apartment. I roamed around in Dubai with him, and we had many discussions during DebConf22 Kosovo. He was the one who took the photo of me on my homepage. I also met him in MiniDebConf22 Palakkad and MiniDebConf23 Tamil Nadu, and went to his flat in Kochi this year in June.

We had many projects in common. He was a Free Software activist and was the designer of the DebConf23 logo, in addition to those for other Debian events in India.

We were all fairly shocked by the news. I was devastated. Food lost its taste, and it became difficult to sleep. That night, Anisa and Kristi cheered me up and gave me company. Thanks a lot to them.

The next day, Joenio also tried to console me. I thank him for doing a great job. I thank everyone who helped me in coping with the difficult situation.

On the next day (the 14th of September), the Debian project leader Jonathan Carter addressed and announced the news officially. THe Debian project also mentioned it on their website.

Abraham was supposed to give a talk, but following the incident, all talks were cancelled for the day. The conference dinner was also cancelled.

As I write, 9 days have passed since his death, but even now I cannot come to terms with it.

Visiting Abraham’s house

On the 15th of September, the conference ran two buses from the hotel to Abraham’s house in Kottayam (2 hours ride). I hopped in the first bus and my mood was not very good. Evangelos (Germany) was sitting opposite me, and he began conversing with me. The distraction helped and I was back to normal for a while. Thanks to Evangelos as he supported me a lot on that trip. He was also very impressed by my use of the StreetComplete app which I was using to edit OpenStreetMap.

In two hours, we reached Abraham’s house. I couldn’t control myself and burst into tears. I went to see the body. I met his family (mother, father and sister), but I had nothing to say and I felt helpless. Owing to the loss of sleep and appetite over the past few days, I had no energy, and didn’t think it was good idea for me to stay there. I went back by taking the bus after one hour and had lunch at the hotel. I withdrew my talk scheduled for the 16th of September.

A Japanese gift

I got a nice Japanese gift from Niibe Yutaka (Japan) - a folder to keep papers which had ancient Japanese manga characters. He said he felt guilty as he swapped his talk with me and so it got rescheduled from 12th September to 16 September which I withdrew later.

Group photo

On the 16th of September, we had a group photo. I am glad that this year I was more clear in this picture than in DebConf22.

Volunteer work and talks attended

I attended the training session for the video team and worked as a camera operator. The Bits from DPL was nice. I enjoyed Abhas’ presentation on home automation. He basically demonstrated how he liberated Internet-enabled home devices. I also liked Kristi’s presentation on ways to engage with the GNOME community.

I also attended lightning talks on the last day. Badri, Wouter, and I gave a demo on how to register on the Prav app. Prav got a fair share of advertising during the last few days.

Departure day

The 18th of September was the day of departure. Badri slept in my room and left early morning (06:30 AM). I dropped him off at the hotel gate. The breakfast was at the eatery (3rd floor) again.

I had an 8 PM flight from Kochi to Delhi, for which I took a cab with Rhonda (Austria), Michael (Nigeria) and Yash (India). We were joined by other DebConf23 attendees at the Kochi airport, where we took another selfie.

Joost and I were on the same flight, and we sat next to each other. He then took a connecting flight from Delhi to Netherlands, while I went with Yash to the New Delhi Railway Station, where we took our respective trains. I reached home on the morning of the 19th of September, 2023.

Big thanks to the organizers

DebConf23 was hard to organize - strict alcohol laws, weird hotel rules, death of a close friend (almost a family member), and a scary notice by the immigration bureau. The people from the team are my close friends and I am proud of them for organizing such a good event.

None of this would have been possible without the organizers who put more than a year-long voluntary effort to produce this. In the meanwhile, many of them had organized local events in the time leading up to DebConf. Kudos to them.

The organizers also tried their best to get clearance for countries not approved by the ministry. I am also sad that people from China, Kosovo, and Iran could not join. In particular, I feel bad for people from Kosovo who wanted to attend but could not (as India does not consider their passport to be a valid travel document), considering how we Indians were so well-received in their country last year.

Note about myself

I am writing this on the 22nd of September, 2023. It took me three days to put up this post - this was one of the tragic and hard posts for me to write. I have literally forced myself to write this. I have still not recovered from the loss of my friend. Thanks a lot to all those who helped me.

PS: Credits to contrapunctus for making grammar, phrasing, and capitalization changes.

]]> Ravi Dwivedi https://workdone0.substack.com/p/cosmos https://workdone0.substack.com/p/cosmos Fri, 15 Sep 2023 00:00:00 GMT Shubham Kumar https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-ide-experience/index.html https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-ide-experience/index.html Thu, 07 Sep 2023 00:00:00 GMT Aditya Athalye programming emacs howto recurse_center clojure https://shrirangkahale.com/posts/education-system-and-youth/ https://shrirangkahale.com/posts/education-system-and-youth/ Wed, 30 Aug 2023 14:20:07 GMT Shrirang Kahale https://aryak.me/blog/04-philosophy-should-be-part-of-edu-system.html https://aryak.me/blog/04-philosophy-should-be-part-of-edu-system.html Sat, 19 Aug 2023 12:39:45 GMT I was recently reading the essay, On Liberty, by John Stuart Mill and I was very intrigued by some of the points brought up by the book.

This led me to think again about something I have been pondering about in and out for ages, the place of philosophy in the educational system.

The problem

I’ll start with the problem. To be honest, this generation is extremely apolitical, and non-politically active.

All they care about is getting good marks, passing 10th/12th and getting a good college/job, while ignoring the big and important question, the state of our nation.

When noone cares about the nation, asatyamev hi jayate (injustice alone will prevail), for there is no person to keep check on the government.

Every person’s goal in life is to serve themselves. Me first, everything secondary. This is how humans work.

The government is meant to mediate this but its not a bulletproof entity, its made of people, whose primary goal is to serve themselves.

This is why we need someone to keep the government in check, and unintellectual and aloofness from politics does not help.

This I feel, is excacerbated by the education system, especially here, which encourages rote learning.

The state of the current education system

This is especially bad with social studies, where it is important to be opinion based, not “I just memorized this chapter and can get 100 in tommorow’s exam”.

I guess this could apply to the sciences, mathematics etc. but it is even more important in the social sciences and humanities.

Social Studies must be opinions, what you think about this thing, who is in the right and who is in the wrong according to you, with multiple sources to understand from.

People should be marked based on the intellectual-ness of the answer, not based on how “politically-correct” or similar to textbook the answer is.

Here is where philosophy plays an important role.

While social studies only covers laws, history and generally how our current system came to be, philosophy questions the basic theory.

Philosophy awakens the intellectual-ness, and makes people think critically.

It encourages and makes people go into deep thoughts and intellectual discussions, which benifit them mentally and society in large as we get more intelligent people.

This encouragement that used to be prevailent in the ancient times of Gurukuls has been effectively wiped off by the advent of the structured educational system.

My ideas for how philosophy could be implemented in education

Philosophy is a vast and varied subject.

I would say, extracts from philosophical texts from the big 4 traditions (western, indian, chinese, islamic) should be given to students about a specific topic, which of course must be suitable for their grade and they should be asked to comment on what they think about each.

In my opinion, everything should be marked based on the intellect of the answer, and when marks are deducted for “wrong-think”, students should be able to appeal to a board setup to answer and co-ordinate with schools about these kind of marking conflicts.

The “untouchable” subjects

There are many subjects that are untouchable in any normal debate, be it caste, morality, religion etc.

This is BAD and leads to forcing of opinions and generally lack of intellectualism around these topics.

You cannot and should not take a blind follow approach to anything. Question every single thing you have to do, be it rhetorical or directed towards someone else.

The law

The law should be more liberal when it comes to “wrong-think”. Speech should be allowed as long as it does not incite or promote violence.

I know incite violence is vague as hell, and that is of course because of how varied stuff can cause people to get mad.

A small social media post about something inciteful can cause more harm than someone saying extremely inciting/dangerous things to even a big audience.

“Hate speech” can not be regulated, and the intentions of something cannot be easily made out.

To be frank, I don’t have any ideas about this. Maybe a committee to decide what constitutes a hate-speech and what does not, but then there are of course biases.

If you go with a simple “if there is violence due to your post we arrest you”, you risk arresting people whose post unintentionally incited the same.

But, at the end of the day, the law should be more allowing for free thinking and intellectual discussions, as long as of course, it doesnt incite violent acts.

Societal stuff

People also are moulded more and more into non-individualistic robots, who all share the same happiness and same opinions.

People like something just because others like it, and if they don’t like it they are treated like outcasts from society.

People have no individualistic goals in life, just the standard “getting good marks, passing 10th/12th and getting a good college/job”

While these are also extremely important, we need to stress on the importance of a life where people aren’t just robots who do what they are told to do (or NPCs as the zoomers call it).

This could be lessened by philosophy, which imbibes critical thinking in the mind of those who read it.

I should have published this on Independence day since well a lot of these were ponderings about the Indian independence and what it gave us Indians, but well procrastination :)

I know this is a bit rambly but I hope you get the point :P

]]> arya@projectsegfau.lt (Arya Kiran) 2023/08/19/6 https://aryak.me/blog/03-e14-linux.html https://aryak.me/blog/03-e14-linux.html Wed, 16 Aug 2023 12:39:45 GMT Network, which I of course disabled. While in Security, I disabled Enhanced Windows Biometric Security (in Virtualization) and Microsoft Device Guard. And thats it for now, I’ll update this blogpost if I face any other issues or discover other stuff. Thanks for sticking around :) If you have any questions, feel free to contact me!]]> I recently got a thinkpad E14 gen5 with a ryzen 5 7th gen to replace my Acer Aspire 7.

Of course I wanted to replace the default windows with linux, but I was too lazy to reinstall and hence just put my old NVMe SSD in the laptop thanks to the availability of 2 M.2 slots.

Before this I had to disable secure boot in the bios which I did along with a few other “important” changes (which I’ll cover later)

Past this, most things worked out of the box after removing my old Nvidia drivers. Ryzen is really good on linux :P

However, 2 things didn’t work correctly, the Fingerprint sensor and WiFi card.

Realtek WiFi

After the initial setup, my realtek wifi card was having frequent disconnection issues. It will suddenly just stop transmitting data and the fix was to reconnect to the wifi network.

Another issue is that the Realtek WiFi card only works with kernels 6.2+. This wasn’t an issue for me as a Debian Sid user but when I first tested the compatibility on an Ubuntu 22.04 ISO before adding my drive, this was pretty weird/confusing.

After a bit of searching, I figured out that the random disconnection was due to NetworkManager’s randomized mac addresses.

To fix this, I just had to append the following to the NetworkManager.conf file:

[device]
wifi.scan-rand-mac-address=no

Goodix Fingerprint

I thought the fingerprint was useless for good, until I discovered that Lenovo provided drivers for their fingerprint sensor through one of the manuals I was searching through.

Originally I didn’t find anything, since the driver page for my model didn’t show linux, but after a bit of searching again, I stumbled across this support page that gave links to drivers.

Since I ran debian testing and its mostly compatible with what Ubuntu has, I tried just installing libfprint-2-2 normally and then the driver’s deb, but the deb refused to install since it needed libfprint-2-tod (a fork of libfprint with support for TOD/touch based fingerprint readers).

I tried installing the deb from ubuntu repos and then install the driver, which succeeded, but didn’t make the device work.

Later, I installed ubuntu 22.04 on a secondary partition to check if it worked on ubuntu, which it did.

One thing I noticed, however was that it used a ppa instead of regular libfprint(-tod) from canonical repos.

Then, I tried installing libfprint and libfprint-tod debs from that ppa and the goodix deb from lenovo, and after a reboot, everything worked!

Once it was up, I installed fprintd and libpam-fprintd, at which point I did the fprintd-enroll and fprintd-verify.

After these succeeded, I ran pam-auth-update --enable fprintd. This made fprintd work on pam stuff like sudo and TTY logins.

With this, the last remaining non-working feature of my thinkpad was working as well :D

The BIOS

Once I got my thinkpad, basically the first thing I did was tweak the BIOS.

Getting into the BIOS is dead simple compared to other laptops, just press enter (which it tells you during bootup) and it will take you to a menu, from where you can choose to go to the BIOS with F1.

I was pleasantly surprised to see a modern BIOS, with touchpad support. Coming from laptops which only had blue and white bioses and barely any tweaks, this was a welcome upgrade :)

First thing to disable was secure boot. It was present in Security section of the BIOS and was pretty easy to disable

Then, I permanently disabled some enterprise spyware utility (UPDATE: it was called Absolute under Security).

There was also Lenovo Cloud Services in Config -> Network, which I of course disabled.

While in Security, I disabled Enhanced Windows Biometric Security (in Virtualization) and Microsoft Device Guard.

And thats it for now, I’ll update this blogpost if I face any other issues or discover other stuff. Thanks for sticking around :)

If you have any questions, feel free to contact me!

]]> arya@projectsegfau.lt (Arya Kiran) 2023/08/16/3 https://ravidwivedi.in/posts/my-steps-transfer-mail-from-gandi/ https://ravidwivedi.in/posts/my-steps-transfer-mail-from-gandi/ Tue, 15 Aug 2023 03:22:43 GMT Earlier this year, gandi.net’s ownership changed and it was acquired by Total Webhosting Solutions. My domain and email were both hosted by this provider. Since I didn’t agree with the ownership change, I decided to switch my email provider. After discussing with Sahil, Snehal, Praveen, Nilesh, I shortlisted two email providers to choose from for ravi at ravidwivedi.in. One was purelymail and the other was mailbox.org. Since purelymail($10 per year) was cheaper than mailbox(€36 per year), I decided to give it a try. If this does not work out, I thought, I will switch to mailbox later.

Below are my steps on how I did it. The steps will be same in principle for any other provider you would like to switch to. So the guide will be helpful to you if you want to switch to any other email provider other than purelymail.

Be sure to read docs of the email provider you want to switch to in addition to this guide.

Step 1: Create an account on purelymail.com

First, I signed up on purelymail and created an account (like something@purelymail.com). They charged $10 upfront for creating this acount.

Step 2: Add your domain to your purelymail account

Sign in to your purelymail account and click on Domains section on the top and you will see a page like the screenshot below:

Click on ‘Add new domain’ as in the above screenshot.

Step 3: Enter domain name

Enter your domain name (like example.com) to the page which you got after clicking ‘Add new domain’ in last step.

Step 4: Create an MX record

Create a new MX record in your gandi.net portal or whatever domain registrar you bought your domain from. Use your mail provider docs to obtain these values and plug in.

Type	Host	Value
MX	(Empty)	mailserver.purelymail.com. (this is a sample value, you should put the value recommended by your email provider. dot at the end of the value is important.)

Step 5: Add an SPF record

Create a new SPF record as you did in Step 4. Again, use your mail provider docs to obtain these values and plug in.

Type	Host	Value
TXT	(Empty)	v=spf1 include:_spf.purelymail.com ~all (this is a sample value, you should put the value recommended by your email provider)

Step 6: Add a TXT record to prove that you own the domain

I am not sure if this step is necessary for other email providers or even for purelymail, but since it was recommended by their docs I added this value from my gandi.net portal. It is a TXT record like this:

Type	Host	Value
TXT	(Empty)	Redacted

Step 7: Add DKIM signatures

DKIM signature is for the email receiver to verify whether the email has indeed been sent by the domain owner. It is highly recommended. You can read more about DKIM signature here. Read your docs of your email provider and put the respective values.

Below is a sample taken from my steps:

Type	Host	Value
CNAME	purelymail1._domainkey	key1.dkimroot.purelymail.com.

Note: the dot at the end is important.

I added two other CNAME records in this step as suggested by purelymail docs.

Step 8: Add DMARC record

Create a DMARC record (check your provider docs for values) as you created other records in previous steps.

Type	Host	Value
CNAME	check your docs	check your docs

Step 9: Check your DNS records

Now we verify whether our records point to our new provider. Purelymail portal has a button ‘Check DNS records’ and it can find whether records you entered are correct. It takes a few minutes and sometimes a few hours for the DNS records to propagate.

Step 10: Create account on new provider

After all the records have been correctly added, create an account on purelymail by clicking ‘Users’ button at the top and create a new user. I created ravi@ user on my domain from there.

Step 11: Import data from gandi.net email

purelymail has an option to import data, like emails, calendar and address book from previous account. Use that option to import emails from gandi.net. Or you can use IMAP to login to your new provider in an email client like Thunderbird and manually copy all the emails to your new account.

Step 12: Delete gandi.net email and DNS records

Delete gandi.net email from the gandi.net portal and remove all the previous records which point to gandi.net for email(like MX, SPF, DKIM).

You are done. Enjoy your new email provider and let me know which one you switched to and why :-)

]]> Ravi Dwivedi https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-getting-about/index.html https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-getting-about/index.html Wed, 02 Aug 2023 00:00:00 GMT Aditya Athalye programming emacs howto recurse_center https://ravidwivedi.in/posts/freebsd-install-issues/ https://ravidwivedi.in/posts/freebsd-install-issues/ Mon, 31 Jul 2023 14:14:19 GMT I installed FreeBSD 13.2 with KDE today on my Dell Inspiron 5482. I have heard good things about FreeBSD primarily as an operating system for servers, but the project claims that it works well for desktop users as well. So, I thought I will give it a try. There were some things I had to figure out during the installation which I will note in this post for future reference. And I had to uninstall FreeBSD within hours of installation, due to issues I faced so it didn’t went well. I am going to list them here and how I figured out some things.

Regdomain selection during install

During install, the installer asked me to select a regdomain. I figured out that the default option – FCC/United States of America worked for me.

Xorg didn’t startup

Post installation, I installed xorg in FreeBSD and added a file /usr/local/etc/X11/xorg.conf.d/20-intel.conf as per the example 1 in the official handbook. But startx command wasn’t working, so I ran the command:

Xorg -configure

and then

cp /root/xorg.conf.new /usr/local/etc/X11/xorg.conf followed by editing the file /usr/local/etc/X11/xorg.conf by changing Driver value to "i1915kms" under the Section “Device”.

Managing brightness during startup

When using FreeBSD, my keyboard’s brightness buttons were not working. To decrease brightness, I used the preinstalled backlight utility.

backlight -f /dev/backlight/backlight0 10

Credits to this answer.

I couldn’t manage the brightness to be automatically low at startup. It was at 100% after very time I booted.

Crashed frequently

The system crashed frequently and wasn’t stabled at all. It automatically reboots after some time. I tried to debug and fix, but couldn’t. So, for now I have uninstalled FreeBSD.

If you know any of the fixes, please let me know.

]]> Ravi Dwivedi https://www.learningfromdata.zingg.ai/p/performance-tuning-snowpark-for-identity https://www.learningfromdata.zingg.ai/p/performance-tuning-snowpark-for-identity Sat, 29 Jul 2023 04:55:12 GMT We have spent the last few weeks on the Zingg Identity Resolution product for Snowflake by battle-testing it for deployment and scale. Built as an entry for the Snowflake Challenge and on top of the Zingg Open Source version, we already had a pretty solid application that could resolve entities at scale within Snowflake. So we did not expect any major hiccups in terms of the overall flow, model building, and prediction. Developer confidence, bolstered by feedback from users you may say ;-) However, since the product leverages Snowpark instead of Spark, we did expect a bit of performance tuning here and there. In this post, I wanted to share the challenges we faced and the changes we made to overcome them. Hope this is useful to others building on Snowflake.

To begin with, we had a 500k test record set running on the x-small size Snowflake warehouse in roughly 12 hours. Though the code was not tuned for Snowflake performance, the job ran successfully to completion without hiccups and we used this as our baseline. Entity Resolution is a tough problem, and harder to do at scale. We want to get maximum mileage out of Snowflake and ensure Zingg ran efficiently for our customers. Our philosophy is that the customer should not have to pay the computing bill for any inefficient code. Hence we always aim to extract maximum performance without throwing more hardware at the problem. Unless we absolutely have to. We also had a target dataset of roughly 2.5 million records, 5 times the baseline. This was a real customer dataset, with non-uniform distribution across fields and with triple the columns of our test set. When we do entity resolution, a 5-fold increase in the input with the same number of columns and match types leads to a 25-fold increase in the comparison complexity. As we have robust index learning based on training data in Zingg along with tons of performance optimizations, we expected the matching job to be somewhere close to 18 hours, which we were planning to tune. However, when we ran this dataset on the x-small Snowflake warehouse, we started seeing errors on session timeout intermittently after roughly 8 hours.

Thanks for reading Sonal’s Newsletter! Subscribe for free to receive new posts and support my work.

com.snowflake.snowpark.SnowparkClientException: Error Code: 0408, Error message: Your Snowpark session has expired. You must recreate your session.
Authentication token has expired. The user must authenticate again.

Running the Snowpark client as a background process or with nohup helped, and we were able to see the job continue to run beyond 24 hours. Surely this performance was not something we wanted to ship. So we killed the run after identifying the first set of bottlenecks. Our approach was to zero in on the hotspots and solve them surgically in the order of highest impact first. The highest impact was mostly about the functionality taking maximum time, but in some cases, we also fixed stuff that was earlier in the flow but slow enough to not let us go past a particular stage in a reasonable time. We used the query profiler by Snowflake extensively for understanding and improving the flow, along with some traces and timing logs in our own code base. The query profiler gives a lot of valuable statistics about the query plan, function timing, and which part of the code triggered the query. It also provides partition-level and row-level information, so it was our go-to source for hotspot identification as well as figuring out if our changes worked as expected. Here is a snapshot of how it looks.

At a high level, here is how the Zingg flow is structured:

Input Data → Index → Join On Index to get Approximate Similar Pairs → Pairwise Similarity Features → Predict Matches → Graph Processing

We learn an index on the input data to create approximately similar pairs. We then compute similarity features on these pairs and predict if the pair is a match or not. This output is clustered using graph algorithms to convert matched pairs to clusters.

Our logs indicated that the biggest bottleneck was the Predict Match flow. The. predict flow leveraged a Snowpark user-defined table function to return two columns. The first was if a given pair was a match or not. The second was the score indicating the similarity of records in the pair with each other. The higher the score, the better the match. Since we returned more than one value from this function, we built it as a UDTF instead of a UDF. However, this UDTF was consuming the bulk of our processing time. Given that we were not really creating multiple rows per input row but returning multiple values per row, all the lateral joins we saw on the query plan did not make sense. It made a worthwhile experiment to convert the UDTF to a user defined function, concatenating the prediction and the score in one value and extracting them out through another UDF. This worked like a charm and immediately cut down the processing on our baseline dataset from 12 hours to 5 and a half hours. More than a 2x improvement! This was a massive gain, but we clearly had more work to do.

We then set out to optimize the queries generated by Snowflake through the Snowpark code we had written. The bottlenecks here were mostly while indexing and generating the near-similar pairs. We saw that if we did computations on a Snowpark Dataframe to build the index and then self-joined the Dataframe on the index, the index computations were repeated. This was counterintuitive to us, as we were using the same instance of the Dataframe. We tried to clone the Dataframe as advised but that did not work. Finally, when we cached the Dataframe, the query plan was altered and our costly computations happened only once. The lesson here was that if the computation is costly and the Dataframe has to be used in a self-join, it is wise to cache the intermediate results. However, this has to be used with caution, as there is an overhead in temporary table creation and disk spillage.

Another optimization we did was on the range scan. We join the records on the computed index to build the pairs. To avoid pairs like Record A, Record B repeat as Record B, Record A, we filter out the pairs on ids. This is critical as the subsequent similarity feature processing is expensive. However, we saw that the join became a cartesian join with filtering pushed out to much later in the flow. This led to slow performance along with a suboptimal flow, making similarity feature computations that we did not need. We cached the Dataframe arising out of the join and changed the id1 > id2 range query to id1-id2 > 0, which pushed the filter at the right junction for our flow as well as changed the earlier cartesian join to a normal join.

Together, the above changes shaved off roughly 20% of runtime on our baseline dataset, and we were now down to 4 hours and 30 minutes of the entity resolution workflow on 500,000 records.

With these major fixes in, we profiled the run again, closely looking through the hotspots, and to our utter dismay, found that the predict function was still the most time-consuming part of the entity resolution workflow. This is a Python UDF that uses machine learning for the prediction. A pre-built model learnt from the training data is passed as an argument to the function along with multiple similarity features of the pair. This UDF predicts whether the pair is a match or not along with their similarity score. Since the features are different for different datasets, we do not know beforehand the shape of the feature columns. Hence we construct an array over the multiple features so that we can resolve different types of entities. In a good bad way, there was no code change we could do to optimize this function. Our code was concise and to the point, and there wasn’t anything to shave off here. So we focused on the Snowflake invocation of this UDF. We learned that Snowflake advocates vectorizing UDFs, especially if they involve matrices and other machine learning workloads. The vectorized UDFs are passed batches of rows, and return batches with results. This made immediate sense as invoking the UDF row by row was surely not going to help with the inherent matrix manipulations. Though we wanted to do this right away, there were quite a few challenges.

It was difficult for us to define the function signature as the column number and types were dynamic. Earlier we used to pass an array, so it had worked. After some thought, we were able to build the signature and define the function on the fly using the metadata we already had while creating the similarity features. Passing the model became tricky however and we were not able to deserialize it in the function no matter what we tried. It seemed there were some extra bytes added when the model was passed as a column received by the UDF as part of a Pandas Dataframe. The same code had been working earlier row by row, so the conversion of Snowframe Dataframe rows to Pandas Dataframe internally seemed to be the likely cause. There was another complexity. We were invoking the Python function through Java Snowpark API, so there were too many layers to unravel here. We went over the open-source code of Snowpark, we tried to add function logging and tracing, we tried to debug calling the UDF from Python, and we read all that we could..we pretty much did anything and everything we could think of!

After struggling for a few anxious days, we changed our approach and began persisting the model to a Snowflake stage. Our UDF would import that location and read and use the model. This finally solved the serde issue we were seeing earlier, along with reducing the data sent to the function. In hindsight, we should have tried this approach first, but at that time the thought was to only change what was absolutely necessary. The code for passing the model had been working for a while, and while tuning performance, it is important to control the changes. So we weren’t too wrong either.

It got a bit late in the day when we ran the build with these changes, and when the test dataset ran in half an hour, we were sure something was crazily wrong. Maybe we had left the bulk of our processing code commented out in the run by mistake? We saw the output table, the results were there, and the predictions looked accurate. It was hard to believe. Too weary to think further, we called it a day.

The next day we double-checked the output and everything was in order. We cross-checked. Yes, it looks fine. No, we are not sure. Not yet. Let us run it once more. Boom, the job was finished by the time we finished our daily catchup + coffee.

28 minutes. From 12 hours. From half a day to half an hour! 24x.

On an x-small warehouse resolving identities across multiple columns for 500,000 rows. Wow! High Five! Now we need a break. Mission Impossible Dead Reckoning, here we come!

It was now time to run the customer data set in their warehouse. Brimming with confidence, we fire the job and monitor it from time to time. Umm..it is not working. The predict function is still a bottleneck. For some reason, the batch size Snowflake has automatically chosen on this dataset is 10, while it was sending 1000 rows at a time to the function on our test data. Possibly because there are way too many columns here compared to our test data? The batch size hint is just a hint, but that seems to be the only ammunition we have right now. Will it work? So we modify the UDF and add the max batch size annotation. This has the desired effect, and we have some great results for our customers.

There were a few other changes we had to make to get here. For example, the Dataframe.show() function would cause an out of memory error on the client machine if the data was big. While working on performance optimization, it is critical that functional parts of the code work flawlessly. We had used show() liberally in our code to verify the logic, assuming that it would limit the records sent to the Snowpark client by limiting rows in the warehouse itself. But show() works differently. It fetches everything to the client and then truncates to 10 rows. This was a bit counterintuitive, but an easy fix.

Overall we are super excited about the gains we have made and so happy that we have come this far. We hope the lessons above will help folks who are building on Snowflake and Snowpark. Feel free to talk to us if you are struggling with Snowpark/Snowflake performance issues, we would love to help and share our learnings! And if you know someone who needs entity resolution on Snowflake, please do send them our way :-)

Ciao!

Thanks for reading Sonal’s Newsletter! Subscribe for free to receive new posts and support my work.

]]> Sonal Goyal https://ravidwivedi.in/posts/self-host-snikket-behind-nginx/ https://ravidwivedi.in/posts/self-host-snikket-behind-nginx/ Tue, 25 Jul 2023 20:23:54 GMT Snikket is a server side software for XMPP chat communications. This guide is for anyone who wants to set up a Snikket server behind nginx. The operating system on the server is debian in my case. My guide is based on the official guide.

Run all the commands mentioned below as root.

Point your domain to server’s IP address

Install curl for debian

apt install curl

Now run:

curl -4 ifconfig.co

The output of the previous command is the IP Address of the server you are using.

Create an A record corresponding to the domain you want to use. Since I have domain ravidwivedi.in, I will deploy my chat server at the subdomain chat.ravidwivedi.in.

Create CNAME records for groups.chat.ravidwivedi.in and share.chat.ravidwivedi.in so that they point to chat.ravidwivedi.in

Open required ports

Open required ports if you are behind a firewall like ufw. The official Snikket docs list which ones you have to allow.

Docker and snikket configuration

Install docker and docker-compose packages on your server. The commands for debian are:

apt install docker docker-compose

Snikket configuration

Create a snikket config file and download docker-compose.yml file prepared by the Snikket project.

mkdir /etc/snikket
cd /etc/snikket
curl -o docker-compose.yml https://snikket.org/service/resources/docker-compose.beta.yml

Create a file named snikket.conf in the /etc/snikket directory with the following contents (replace appropriate fields according to your domain and email):

# The primary domain of your Snikket instance
SNIKKET_DOMAIN=chat.ravidwivedi.in

# An email address where the admin can be contacted
# (also used to register your Let's Encrypt account to obtain certificates)
SNIKKET_ADMIN_EMAIL=your-email@ravidwivedi.in

Nginx config and HTTPS

For nginx setup, we follow Snikket project’s reverse proxy docs. We will also setup HTTPS certificates using certbot. Let’s install nginx and certbot:

apt install nginx python3-certbot-nginx

Obtain certificates for all the subdomains

certbot certonly --standalone -d share.chat.ravidwivedi.in -d groups.chat.ravidwivedi.in  -d chat.ravidwivedi.in

The output for me was:

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/chat.ravidwivedi.in/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/chat.ravidwivedi.in/privkey.pem
This certificate expires on 2023-10-23.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

We want nginx to listen at ports 80 and 443. So, let’s direct Snikket to bind to ports 5080 and 5443 to avoid conflict with nginx. To do this, add these lines to /etc/snikket/snikket.conf:

SNIKKET_TWEAK_HTTP_PORT=5080
SNIKKET_TWEAK_HTTPS_PORT=5443

Now create a file /etc/nginx/sites-available/chat.ravidwivedi.in and add the following contents to it, followed by replacing ravidwivedi.in with your domain name and specifying correct ssl_certificate path to the location of your certificates.

server {
  # Accept HTTP connections
  listen 80;
  listen [::]:80;

  server_name chat.ravidwivedi.in;
  server_name groups.chat.ravidwivedi.in;
  server_name share.chat.ravidwivedi.in;

  location / {
      proxy_pass http://localhost:5080/;
      proxy_set_header      Host              $host;
      proxy_set_header      X-Forwarded-For   $proxy_add_x_forwarded_for;

      # This is the maximum size of uploaded files in Snikket
      client_max_body_size 104857616; # 100MB + 16 bytes
  }
}

server {
  # Accept HTTPS connections
  listen [::]:443 ssl ipv6only=on;
  listen 443 ssl;
  ssl_certificate /path/to/certificate.pem;
  ssl_certificate_key /path/to/key.pem;

  server_name chat.ravidwivedi.in;
  server_name groups.chat.ravidwivedi.in;
  server_name share.chat.ravidwivedi.in;

  location / {
      proxy_pass https://localhost:5443/;
      proxy_set_header      Host              $host;
      proxy_set_header      X-Forwarded-For   $proxy_add_x_forwarded_for;
      # REMOVE THIS IF YOU CHANGE `localhost` TO ANYTHING ELSE ABOVE
      proxy_ssl_verify      off;
      proxy_set_header      X-Forwarded-Proto https;
      proxy_ssl_server_name on;

      # This is the maximum size of uploaded files in Snikket
      client_max_body_size 104857616; # 100MB + 16 bytes

      # For BOSH and WebSockets
      proxy_set_header Connection $http_connection;
      proxy_set_header Upgrade $http_upgrade;
      proxy_read_timeout 900s;

  }
}

In the above config, we need to specify path to our certificates created by certbot in an earlier step of this guide. Locate the following lines:

ssl_certificate /path/to/certificate.pem;
ssl_certificate_key /path/to/key.pem;

The output of my certbot command earlier told me that the certificates are at /etc/letsencrypt/live/chat.ravidwivedi.in/fullchain.pem and ssl_key is at /etc/letsencrypt/live/chat.ravidwivedi.in/privkey.pem

So we change the above mentioned two lines in the nginx config at /etc/nginx/sites-available/chat.ravidwivedi.in to:

ssl_certificate /etc/letsencrypt/live/chat.ravidwivedi.in/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/chat.ravidwivedi.in/privkey.pem;

After making these changes to file, save it. Now, create symlink to /etc/nginx/sites-enabled/chat.ravidwivedi.in file by running the command:

ln -s /etc/nginx/sites-available/chat.ravidwivedi.in /etc/nginx/sites-enabled/chat.ravidwivedi.in

Restart nginx:

systemctl restart nginx

You can also check if nginx syntax is correct by running

nginx -t

If the output of this command indicates an error, you need to fix that error before moving on.

Launch Snikket

Go back into /etc/snikket directory by running:

cd /etc/snikket

Now run:

docker-compose up -d

After the command run is complete, visit chat.ravidwivedi.in and you will see a login page as in the screenshot below.

If you don’t see this login page, there is some error in your setup.

If you are able to see that login page, create an account with admin privileges by running the command:

docker exec snikket create-invite --admin --group default

Post install

Snikket project has a page mentioning additional lines you can add to your snikket config file, like for example, setting limit on size of each attachment. Also, check out the upgrade page on how to keep snikket software updated.

If you are using Snikket, be sure to donate to the Snikket project to support them.

]]> Ravi Dwivedi https://mrkaran.dev/posts/nomad-k8s-showdown/ https://mrkaran.dev/posts/nomad-k8s-showdown/ Sun, 23 Jul 2023 05:34:47 GMT This blog post is ignited by the following Twitter exchange:

I don’t take the accusation of unsubstantiated argument, especially on a technical topic lightly. I firmly believe in substantiated arguments and hence, here I am, elaborating on my stance. If found mistaken, I am open to corrections and revise my stance.

Some Historical Context#

In my professional capacity, I have run and managed several K8s clusters (using AWS EKS) for our entire team of devs (been there done that). The most complex piece of our otherwise simple and clean stack was K8s and we’d been longing to find a better replacement. None of us knew whether that would be Nomad or anything else. But we took the chance and we have reached a stage where we can objectively argue that, for our specific workloads, Nomad has proven to be a superior tool compared to K8s.

Building with Nomad#

Nomad presents a fundamental building block approach to designing your own services. It used to be true that Nomad was primarily a scheduler, and for serious production workloads, you had to rely on Consul for service discovery and Vault for secret management. However, this scenario has changed as Nomad now seamlessly integrates these features, making them first-class citizens in its environment. Our team replaced our HashiCorp stack with just Nomad, and we never felt constrained in terms of what we could accomplish with Consul/Vault. While these tools still hold relevance for larger clusters managed by numerous teams, they are not necessary for our use case.

Deconstructing Infrastructure#

Kubernetes employs a declarative state for every operation in the cluster, essentially operating as a reconciliation mechanism to keep everything in check. In contrast, Nomad requires dealing with fewer components, making it appear lacking compared to K8s’s concept of everything being a “resource.” However, that is far from the truth.

• Ingress: We run a set of HAProxy on a few nodes which act as “L7 LBs”. Configured with Nomad services, they can do the routing based on Host headers.
• DNS: To provide external access to a service without using a proxy, we developed a tool that scans all services registered in the cluster and creates a corresponding DNS record on AWS Route53.
• Monitoring: Ah my fav. You wanna monitor your K8s cluster. Sure, here’s kube-prometheus, prometheus-operator, kube-state-metrics. Choices, choices. Enough to confuse you for days. Anyone who’s ever deployed any of these, tell me why this thing needs such a monstrosity setup of CRDs and operators. Monitoring Nomad is such a breeze, 3 lines of HCL config and done.
• Statefulsets: It’s 2023 and the irony is rich - the recommended way to run a database inside K8s is… not to run it inside K8s at all. In Nomad, we run a bunch of EC2 instances and tag them as db nodes. The DBs don’t float around as containers to random nodes. And there’s no CSI plugin reaching for a storage disk in AZ-1 when the node is basking in AZ-2. Running a DB on Nomad feels refreshingly like running it on an unadorned EC2 instance.
• Autoscale: All our client nodes (except for the db nodes) are ephemeral and part of AWS’s Auto Scaling Groups (ASGs). We use ASG rules for the horizontal scaling of the cluster. While Nomad does have its own autoscale, our preference is to run large instances dedicated to specific workloads, avoiding a mix of different workloads on the same machine.

Over abstraction of Kubernetes#

One of my primary critiques of K8s is its hidden complexities. While these abstractions might simplify things on the surface, debugging becomes a nightmare when issues arise. Even after three years of managing K8s clusters, I’ve never felt confident dealing with databases or handling complex networking problems involving dropped packets.

You might argue that it’s about technical chops, which I won’t disagree with - but then do you want to add value to the business by getting shit done or do you want to be the resident K8s whiz at your organization?

Consider this: How many people do you know who run their own K8s clusters? Even the K8s experts themselves preach about running prod clusters on EKS/GKE etc. How many fully leverage all that K8s has to offer? How many are even aware of all the network routing intricacies managed by kube-proxy? If these queries stir up clouds of uncertainty, it’s possible you’re sipping the Kubernetes Kool-Aid without truly comprehending the recipe, much like I found myself doing at one point

Nomad: Not Perfect, But Simpler#

Now, if you’re under the impression that I’m singing unabashed praises for Nomad, let me clarify - Nomad has its share of challenges. I’ve personally encountered and reported several. However, the crucial difference lies in Nomad’s lesser degree of abstraction, allowing for a comprehensive understanding of its internals. For instance, we encountered service reconciliation issues with a particular Nomad version. However, we could query the APIs, identify the problem, and write a bash script to resolve and reconcile it. It wouldn’t have been possible when there are too many moving parts in the system and we don’t know where to even begin debugging.

The YAML hell is all too well known to all of us. In K8s, writing job manifests required a lot of effort (by the developers who don’t work with K8s all day) and were very complex to understand. It felt “too verbose” and involved copy pasting large blocks from the docs and trying to make things work. Compare that to HCL, it feels much nicer to read and shorter. Things are more straightforward to understand.

I’ve not even touched upon the nice-ities on Nomad yet. Like better humanly understandable ACLs? Cleaner and simpler job spec, which defines the entire job in one file? A UI which actually shows everything about your cluster, nodes, and jobs? Not restricting your workloads to be run as Docker containers? A single binary which powers all of this?

The central question this post aims to raise is: What can K8s do that Nomads can’t, especially considering the features people truly need? My perspectives are informed not only by my organization but also through interactions with several other organizations at various meetups and conferences. Yet, I have rarely encountered a use case that could only be managed by K8s. While Nomad isn’t a panacea for all issues, it’s certainly worth a try. Reducing the complexity of your tech stack can prove beneficial for your applications and, most importantly, your developers.

At this point, K8s enjoys immense industry-wide support, while Nomad remains the unassuming newcomer. This contrast is not a negative aspect, per se. Large organizations often gravitate towards complexity and the opportunity to engage more engineers. However, if simplicity were the primary goal, the prevailing sense of overwhelming complexity in the infrastructure and operations domain wouldn’t be as pervasive.

Conclusion#

I hope my arguments provide a more comprehensive perspective and address the earlier critique of being unsubstantiated.

Update#

INFO

Darren has responded to this blog post. You can read the response on Twitter.

Fin!

]]> Karan Sharma https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-midway-refactor/index.html https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-midway-refactor/index.html Sun, 23 Jul 2023 00:00:00 GMT Aditya Athalye programming emacs howto recurse_center https://mrkaran.dev/posts/storing-aws-pinpoint-logs/ https://mrkaran.dev/posts/storing-aws-pinpoint-logs/ Fri, 21 Jul 2023 05:24:06 GMT Kinesis Firehose -> Vector HTTP -> Clickhouse Ingesting Data# On the HTTP server side, we used vector’s aws_kinesis_firehose source. Compared to just using http source, here are the differences I found: Has first-class support for access_key. AWS Kinesis can be configured to send access_key which comes as the value X-Amz-Firehose-Access-Key header in the HTTP request. This means that the request which contains an invalid access key will be rejected at the source itself. However, in the http source, I couldn’t find a way to drop events at the source level. It is required to use a VRL transformer to check whether X-Amz-Firehose-Access-Key is present in the headers and do a value comparison with our key. Has native support for base64 decoding the payload. This one’s pretty useful and saved me a lot of VRL transformer rules that I would have otherwise written with the http source. So, basically, this is how the server receives the payload: { "requestId": "6a14a06b-6eae-4218-...", "timestamp": 1689766125971, "records": [ { "data": "eyJld..." }, { "data": "eyJldmVudF9..." } ] } The value of the payload is a base64 encoded value of the JSON Object of an SMS event. However, the aws_kinesis_firehose source is smart enough and automagically decodes this list of records and their values into individual events. This is how the final event looks like when using aws_kinesis_firehose source: { "message": "{\"event_type\":\"_SMS.SUCCESS\",\"event_timestamp\":1689827914426,\"arrival_timestamp\":1689827917659,\"event_version\":\"3.1\",\"application\":{\"app_id\":\"redacted\",\"sdk\":{}},\"client\":{\"client_id\":\"redacted\"},\"device\":{\"platform\":{}},\"session\":{},\"attributes\":{\"sender_request_id\":\"redacted\",\"destination_phone_number\":\"+91xxx\",\"record_status\":\"DELIVERED\",\"iso_country_code\":\"IN\",\"mcc_mnc\":\"xxx\",\"number_of_message_parts\":\"1\",\"message_id\":\"redacted\",\"message_type\":\"Transactional\",\"origination_phone_number\":\"redactedORG\"},\"metrics\":{\"price_in_millicents_usd\":xx.0},\"awsAccountId\":\"redacted\"}\n", "request_id": "6dd45388-xxx", "source_arn": "arn:aws:firehose:ap-south-1:redacted:deliverystream/redacted", "source_type": "aws_kinesis_firehose", "timestamp": "2023-07-20T04:39:38.772Z" } This makes it straightforward because now we just have to parse the JSON inside the message key and do transformations on that object. If it was http source, then I’d to loop over the records structure and figure out how to split them as individual events for the rest of the Vector pipeline… which would have been messy to say the least. Here’s the vector config so far: [sources.firehose] # General type = "aws_kinesis_firehose" address = "127.0.0.1:9000" store_access_key = false access_keys = ["superdupersecret"] # Use it for debugging [sinks.console] type = "firehose" inputs = ["format_pinpoint_logs"] encoding.codec = "json" Formatting the data# Now that we have a pipeline which sends and receives data, we can process the events and transform them into a schema that is more desirable. Since we require the events to be queryable in a Clickhouse DB, this is the schema we have: CREATE TABLE default.pinpoint_logs ( `_timestamp` DateTime('Asia/Kolkata'), `app_id` LowCardinality(String), `event_type` LowCardinality(String), `record_status` LowCardinality(String), `origination_phone_number` String, `message_id` String, `destination_phone_number` String, `arrival_timestamp` DateTime('Asia/Kolkata'), `event_timestamp` DateTime('Asia/Kolkata'), `meta` Nullable(String) ) ENGINE = MergeTree PARTITION BY toYYYYMM(_timestamp) ORDER BY _timestamp SETTINGS index_granularity = 8192; To achieve the above format, we can use VRL to parse and format our SMS events: [transforms.format_pinpoint_logs] type = "remap" inputs = ["firehose"] source = ''' # Decode the JSON message and set ingestion timestamp .message = parse_json!(.message) .ingestion_timestamp = .timestamp # Convert timestamps from Unix to DateTime .event_timestamp = from_unix_timestamp!(.message.event_timestamp, unit:"milliseconds") .arrival_timestamp = from_unix_timestamp!(.message.arrival_timestamp, unit:"milliseconds") # Extract keys to top level and remove from attributes .record_status = del(.message.attributes.record_status) .origination_phone_number = del(.message.attributes.origination_phone_number) .destination_phone_number = del(.message.attributes.destination_phone_number) .message_id = del(.message.attributes.message_id) # Encode the remaining attributes as JSON string .attr = encode_json(.message.attributes) # Format Payload for Clickhouse . = { "_timestamp": .ingestion_timestamp, "arrival_timestamp": .arrival_timestamp, "event_timestamp": .event_timestamp, "app_id": .message.application.app_id, "event_type": .message.event_type, "record_status": .record_status, "message_id": .message_id, "origination_phone_number": .origination_phone_number, "destination_phone_number": .destination_phone_number, "meta": .attr } ''' Plugging this, we have a clean JSON object for each SMS event. The only thing now we need to add is an output sink to Clickhouse: [sinks.clickhouse] type = "clickhouse" inputs = ["format_pinpoint_logs"] skip_unknown_fields = true compression = "gzip" database = "default" endpoint = "http://127.0.0.1:8123" table = "pinpoint_logs" encoding.timestamp_format = "unix" batch.max_bytes = 1049000 # 1 MB batch.timeout_secs = 5 buffer.max_size = 268435488 buffer.type = "disk" buffer.when_full = "block" Perfect! On running this pipeline with vector -c config.toml we can see the consumption the records Hope this short post was useful if you’ve to do anything similar! Fin!]]> At $dayjob, we use AWS Pinpoint to send out SMS to our customers. We’ve also written a detailed blog post on how we use Clickhouse + vector stack for our logging needs. We additionally wanted to store the delivery logs generated by the Pinpoint service. But like with anything else in AWS, even simpler tasks like these usually tend to piggyback on other counterparts of AWS - in this case, it happens to be AWS Kinesis. All the delivery logs which contain metadata about SMS delivery are streamed to Kinesis.

Our setup involves configuring Pinpoint with Amazon Kinesis Data Firehose stream. Firehose is an ETL service that helps stream events to other persistent stores. Firehose supports multiple such output sinks and in our case we use HTTP sink.

This is what the flow looks like:

Pinpoint -> Kinesis Firehose -> Vector HTTP -> Clickhouse

---

Ingesting Data#

On the HTTP server side, we used vector’s aws_kinesis_firehose source. Compared to just using http source, here are the differences I found:

• Has first-class support for access_key. AWS Kinesis can be configured to send access_key which comes as the value X-Amz-Firehose-Access-Key header in the HTTP request. This means that the request which contains an invalid access key will be rejected at the source itself. However, in the http source, I couldn’t find a way to drop events at the source level. It is required to use a VRL transformer to check whether X-Amz-Firehose-Access-Key is present in the headers and do a value comparison with our key.

• Has native support for base64 decoding the payload. This one’s pretty useful and saved me a lot of VRL transformer rules that I would have otherwise written with the http source. So, basically, this is how the server receives the payload:

  {
    "requestId": "6a14a06b-6eae-4218-...",
    "timestamp": 1689766125971,
    "records": [
        {
            "data": "eyJld..."
        },
        {
            "data": "eyJldmVudF9..."
        }
    ]
  }

  The value of the payload is a base64 encoded value of the JSON Object of an SMS event. However, the aws_kinesis_firehose source is smart enough and automagically decodes this list of records and their values into individual events. This is how the final event looks like when using aws_kinesis_firehose source:

      {
        "message": "{\"event_type\":\"_SMS.SUCCESS\",\"event_timestamp\":1689827914426,\"arrival_timestamp\":1689827917659,\"event_version\":\"3.1\",\"application\":{\"app_id\":\"redacted\",\"sdk\":{}},\"client\":{\"client_id\":\"redacted\"},\"device\":{\"platform\":{}},\"session\":{},\"attributes\":{\"sender_request_id\":\"redacted\",\"destination_phone_number\":\"+91xxx\",\"record_status\":\"DELIVERED\",\"iso_country_code\":\"IN\",\"mcc_mnc\":\"xxx\",\"number_of_message_parts\":\"1\",\"message_id\":\"redacted\",\"message_type\":\"Transactional\",\"origination_phone_number\":\"redactedORG\"},\"metrics\":{\"price_in_millicents_usd\":xx.0},\"awsAccountId\":\"redacted\"}\n",
        "request_id": "6dd45388-xxx",
        "source_arn": "arn:aws:firehose:ap-south-1:redacted:deliverystream/redacted",
        "source_type": "aws_kinesis_firehose",
        "timestamp": "2023-07-20T04:39:38.772Z"
    }

  This makes it straightforward because now we just have to parse the JSON inside the message key and do transformations on that object. If it was http source, then I’d to loop over the records structure and figure out how to split them as individual events for the rest of the Vector pipeline… which would have been messy to say the least.

Here’s the vector config so far:

[sources.firehose]
# General
type = "aws_kinesis_firehose"
address = "127.0.0.1:9000"
store_access_key = false
access_keys = ["superdupersecret"]

# Use it for debugging
[sinks.console]
type = "firehose"
inputs = ["format_pinpoint_logs"]
encoding.codec = "json"

Formatting the data#

Now that we have a pipeline which sends and receives data, we can process the events and transform them into a schema that is more desirable. Since we require the events to be queryable in a Clickhouse DB, this is the schema we have:

CREATE TABLE default.pinpoint_logs (
    `_timestamp` DateTime('Asia/Kolkata'),
    `app_id` LowCardinality(String),
    `event_type` LowCardinality(String),
    `record_status` LowCardinality(String),
    `origination_phone_number` String,
    `message_id` String,
    `destination_phone_number` String,
    `arrival_timestamp` DateTime('Asia/Kolkata'),
    `event_timestamp` DateTime('Asia/Kolkata'),
    `meta` Nullable(String)
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(_timestamp)
ORDER BY _timestamp
SETTINGS index_granularity = 8192;

To achieve the above format, we can use VRL to parse and format our SMS events:

[transforms.format_pinpoint_logs]
type = "remap" 
inputs = ["firehose"] 
source = '''
  # Decode the JSON message and set ingestion timestamp
  .message = parse_json!(.message)
  .ingestion_timestamp = .timestamp

  # Convert timestamps from Unix to DateTime
  .event_timestamp = from_unix_timestamp!(.message.event_timestamp, unit:"milliseconds")
  .arrival_timestamp = from_unix_timestamp!(.message.arrival_timestamp, unit:"milliseconds")

  # Extract keys to top level and remove from attributes
  .record_status = del(.message.attributes.record_status)
  .origination_phone_number = del(.message.attributes.origination_phone_number)
  .destination_phone_number = del(.message.attributes.destination_phone_number)
  .message_id = del(.message.attributes.message_id)

  # Encode the remaining attributes as JSON string
  .attr = encode_json(.message.attributes)

  # Format Payload for Clickhouse
  . = {
    "_timestamp": .ingestion_timestamp,
    "arrival_timestamp": .arrival_timestamp,
    "event_timestamp": .event_timestamp,
    "app_id": .message.application.app_id,
    "event_type": .message.event_type,
    "record_status": .record_status,
    "message_id": .message_id,
    "origination_phone_number": .origination_phone_number,
    "destination_phone_number": .destination_phone_number,
    "meta": .attr
  }
'''

Plugging this, we have a clean JSON object for each SMS event. The only thing now we need to add is an output sink to Clickhouse:

[sinks.clickhouse]
type = "clickhouse"
inputs = ["format_pinpoint_logs"]
skip_unknown_fields = true
compression = "gzip"
database = "default"
endpoint = "http://127.0.0.1:8123"
table = "pinpoint_logs"
encoding.timestamp_format = "unix"
batch.max_bytes = 1049000 # 1 MB
batch.timeout_secs = 5
buffer.max_size = 268435488
buffer.type = "disk"
buffer.when_full = "block"

Perfect! On running this pipeline with vector -c config.toml we can see the consumption the records

Hope this short post was useful if you’ve to do anything similar!

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/bridge-network-in-nomad/ https://mrkaran.dev/posts/bridge-network-in-nomad/ Mon, 17 Jul 2023 05:45:04 GMT mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0@if113: mtu 1500 qdisc noqueue state UP group default link/ether 76:47:6d:49:00:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.26.64.5/20 brd 172.26.79.255 scope global eth0 valid_lft forever preferred_lft forever We can see that one end of the pair is eth0 (container’s default gateway) which is connected to a network interface with an index 113. For the tunnel to actually work, the veth pair should also exist on the host: $ ip a 113: veth3402deda@if2: mtu 1500 qdisc noqueue master nomad state UP group default link/ether 3a:85:1b:37:75:17 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::3885:1bff:fe37:7517/64 scope link valid_lft forever preferred_lft forever So, when we see veth3402deda@if2 in the host’s network namespace (with the index 113), and then we see eth0@if113 inside the Redis container, we can infer that these two interfaces form a veth pair: veth3402deda@if2 on the host side and eth0 inside the container. This connection enables the container to communicate with the external network through the host’s network stack. Capturing packets# We can capture TCP packets on the veth interface to see the routing work: sudo tcpdump -i veth971858d5 -n tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on veth971858d5, link-type EN10MB (Ethernet), snapshot length 262144 bytes 10:51:27.801319 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [S], seq 1331933249, win 65495, options [mss 65495,sackOK,TS val 248300785 ecr 0,nop,wscale 7], length 0 10:51:27.801549 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [S.], seq 107697826, ack 1331933250, win 65160, options [mss 1460,sackOK,TS val 3965422857 ecr 248300785,nop,wscale 7], length 0 10:51:27.801616 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 1, win 512, options [nop,nop,TS val 248300785 ecr 3965422857], length 0 10:51:27.801737 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [P.], seq 1:79, ack 1, win 512, options [nop,nop,TS val 248300786 ecr 3965422857], length 78 10:51:27.801751 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [.], ack 79, win 509, options [nop,nop,TS val 3965422858 ecr 248300786], length 0 10:51:27.802022 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [P.], seq 1:4097, ack 79, win 509, options [nop,nop,TS val 3965422858 ecr 248300786], length 4096 10:51:27.802059 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 4097, win 491, options [nop,nop,TS val 248300786 ecr 3965422858], length 0 10:51:27.802120 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [P.], seq 4097:5396, ack 79, win 509, options [nop,nop,TS val 3965422858 ecr 248300786], length 1299 10:51:27.802135 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 5396, win 502, options [nop,nop,TS val 248300786 ecr 3965422858], length 0 10:51:27.803484 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [F.], seq 79, ack 5396, win 512, options [nop,nop,TS val 248300787 ecr 3965422858], length 0 10:51:27.803567 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [F.], seq 5396, ack 80, win 509, options [nop,nop,TS val 3965422859 ecr 248300787], length 0 10:51:27.803597 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 5397, win 512, options [nop,nop,TS val 248300787 ecr 3965422859], length 0 10:53:08.523431 IP 172.26.64.6.53042 > 95.216.165.210.27372: Flags [.], ack 2169295212, win 501, options [nop,nop,TS val 735542538 ecr 4133067854], length 0 10:53:08.523551 IP 95.216.165.210.27372 > 172.26.64.6.53042: Flags [.], ack 1, win 509, options [nop,nop,TS val 4133379150 ecr 735231242], length 0 10:53:08.523554 IP 95.216.165.210.27372 > 172.26.64.6.53042: Flags [.], ack 1, win 509, options [nop,nop,TS val 4133379150 ecr 735231242], length 0 10:53:08.523562 IP 172.26.64.6.53042 > 95.216.165.210.27372: Flags [.], ack 1, win 501, options [nop,nop,TS val 735542538 ecr 4133379150], length 0 To summarize the output, we can see that the log is showing a TCP connection between 172.26.64.1 (source) and 172.26.64.6 (destination), specifically on port 7000. 172.26.64.1 happens to be the gateway for nomad subnet. Summary# Hope this post clarified some networking internals and behind the scenes magic when using Nomad bridge networking. Refer to my other post - Nomad networking explained for a practical breakdown of all the different ways to expose and connect applications in a Nomad cluster. Fin!]]> To set the stage, it’s crucial to understand what we mean by “bridge networking”. In a nutshell, it is a type of network connection in Linux that allows virtual interfaces, like the ones used by virtual machines and containers, to share a physical network interface.

With Nomad, when a task is allocated, it creates a network namespace with its own network stack. Within this, a virtual ethernet (veth) pair is established, one end of which is assigned to the network namespace of the allocation, and the other remains in the host namespace.

The Network Journey#

To illustrate this practically, let’s assume a packet is sent from a task within an allocation. The packet would first be received by the local end of the veth pair, it would then traverse to the other end residing in the host’s namespace. From there, it is sent to the bridge on the host (in this case, the “nomad” bridge), which finally sends the packet out to the world via the host’s physical network interface (typically “eth0” or equivalent in your machine).

The journey of a packet from the outside world to a task inside an allocation is the exact mirror image. The packet reaches “eth0” first, then the nomad bridge, it is then forwarded to the appropriate veth interface in the host’s namespace. From there, it crosses over to the other end of the veth pair in the allocation’s network namespace and finally gets routed to the destination task.

To bridge or not to#

Let’s take a look at the following jobspec which is for deploying my tiny side project - Cloak on Nomad

job "cloak" {
  datacenters = ["dc1"]
  type        = "service"

  group "redis" {
    network {
      mode = "host"
      port "redis" {
        to = 6379
      }
    }

    service {
      name     = "cloak-redis"
      port     = "redis"
      provider = "nomad"
    }


    task "redis" {
      driver = "docker"


      config {
        image                  = "redis:7"
        advertise_ipv6_address = false

        ports = [
          "redis",
        ]

        volumes = [
          "/data/cloak/redis:/data",
        ]
      }

      resources {
        cpu    = 500 # MHz
        memory = 256 # MB
      }
    }
  }

  group "cloak" {
    network {
      mode = "host"
      port "cloak" {
        static = 7000
        to     = 7000
      }
    }

    task "cloak" {
      driver = "docker"

      config {
        image   = "ghcr.io/mr-karan/cloak:v0.2.0"
        command = "--config=config.toml"
        ports = [
          "cloak",
        ]
      }

      template {
        data        = <<EOH
# Configuration for 1 redis instances, as assigned via rendezvous hashing.
{{$allocID := env "NOMAD_ALLOC_ID" -}}
{{range nomadService 1 $allocID "cloak-redis"}}
CLOAK_REDIS__address={{ .Address }}:{{ .Port }}
{{- end}}
EOH
        destination = "secrets/file.env"
        env         = true
      }


      resources {
        cpu    = 500 # MHz
        memory = 700 # MB
      }
    }
  }
}

Our focus should be on the network.mode stanza. To illustrate what happens behind the scenes when an alloc runs in network.mode=host (host network), we can run the above job.

On the machine, we can see that port 7000 (static) and port 27042 (dynamic) are allocated on the host network interface (eth0):

We can also see the port and process details using ss:

sudo ss -ltpn 'sport = :7000'
State    Recv-Q   Send-Q      Local Address:Port       Peer Address:Port   Process
LISTEN 0 4096      95.216.165.210:7000    0.0.0.0:*  users:(("docker-proxy",pid=67068,fd=4))

This config is more suitable for specific workloads - like load balancers or similar deployments where you want to expose the network interface on the host. It’s also helpful for applications running outside of Nomad on that host to connect via the host network interface.

However, typically in a job where you want to connect to multiple different allocs - you’d want to set up a bridge network. This generally avoids exposing the workload on the host network directly. It’s a typical setup where you want to put applications behind a reverse proxy (NGINX/Caddy).

Let’s change network.mode=bridge in the above job spec and see the changes.

$ nomad job plan cloak.nomad

+/- Job: "cloak"
+/- Task Group: "cloak" (1 create/destroy update)
  + Network {
      Hostname: ""
    + MBits:    "0"
    + Mode:     "bridge"
    + Static Port {
      + HostNetwork: "default"
      + Label:       "cloak"
      + To:          "7000"
      + Value:       "7000"
      }
    }
  - Network {
      Hostname: ""
    - MBits:    "0"
    - Mode:     "host"
    - Static Port {
      - HostNetwork: "default"
      - Label:       "cloak"
      - To:          "7000"
      - Value:       "7000"
      }
    }
    Task: "cloak"

+/- Task Group: "redis" (1 create/destroy update)
  + Network {
      Hostname: ""
    + MBits:    "0"
    + Mode:     "bridge"
    + Dynamic Port {
      + HostNetwork: "default"
      + Label:       "redis"
      + To:          "6379"
      }
    }
  - Network {
      Hostname: ""
    - MBits:    "0"
    - Mode:     "host"
    - Dynamic Port {
      - HostNetwork: "default"
      - Label:       "redis"
      - To:          "6379"
      }
    }
    Task: "redis"

Now we don’t see the ports forwarded on the host network:

Similarly, ss also shows no process listening on the host network

IPTables and Routing#

To understand what happened when we switched the networking mode to bridge, we need to take a look at the Nomad iptables magic which comes into play when using bridge network.

I pulled up the iptables and saw specific rules under the chains CNI-FORWARD and NOMAD-ADMIN. These rules, in essence, allow all traffic to and from the allocation’s network namespace.

$ sudo iptables -L CNI-FORWARD
Chain CNI-FORWARD (1 references)
target     prot opt source               destination         
NOMAD-ADMIN  all  --  anywhere             anywhere             /* CNI firewall plugin admin overrides */
ACCEPT     all  --  anywhere             172.26.64.5          ctstate RELATED,ESTABLISHED
ACCEPT     all  --  172.26.64.5          anywhere            
ACCEPT     all  --  anywhere             172.26.64.6          ctstate RELATED,ESTABLISHED
ACCEPT     all  --  172.26.64.6          anywhere

sudo iptables -L NOMAD-ADMIN
Chain NOMAD-ADMIN (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             172.26.64.0/20

Nomad uses 172.26.64.0/20 as the default subnet for the bridge network. The IPs 172.26.64.5 and 172.26.64.6 are assigned to 2 different allocs in this CIDR. The iptables rules allow complete traffic to flow on this subnet.

To check the routing,ip route command can be used.

$ ip route show 172.26.64.0/20
172.26.64.0/20 dev nomad proto kernel scope link src 172.26.64.1

It uses the nomad network interface for routing packets related to the default bridge network.

Using nsenter we can find more details about the network namespace created for an alloc. Let’s find details about the redis alloc:

sudo nsenter -t $(pgrep redis) --net ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0@if113: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 76:47:6d:49:00:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.26.64.5/20 brd 172.26.79.255 scope global eth0
       valid_lft forever preferred_lft forever

We can see that one end of the pair is eth0 (container’s default gateway) which is connected to a network interface with an index 113. For the tunnel to actually work, the veth pair should also exist on the host:

$ ip a
113: veth3402deda@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master nomad state UP group default 
    link/ether 3a:85:1b:37:75:17 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::3885:1bff:fe37:7517/64 scope link 
       valid_lft forever preferred_lft forever

So, when we see veth3402deda@if2 in the host’s network namespace (with the index 113), and then we see eth0@if113 inside the Redis container, we can infer that these two interfaces form a veth pair: veth3402deda@if2 on the host side and eth0 inside the container. This connection enables the container to communicate with the external network through the host’s network stack.

Capturing packets#

We can capture TCP packets on the veth interface to see the routing work:

sudo tcpdump -i veth971858d5 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on veth971858d5, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:51:27.801319 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [S], seq 1331933249, win 65495, options [mss 65495,sackOK,TS val 248300785 ecr 0,nop,wscale 7], length 0
10:51:27.801549 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [S.], seq 107697826, ack 1331933250, win 65160, options [mss 1460,sackOK,TS val 3965422857 ecr 248300785,nop,wscale 7], length 0
10:51:27.801616 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 1, win 512, options [nop,nop,TS val 248300785 ecr 3965422857], length 0
10:51:27.801737 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [P.], seq 1:79, ack 1, win 512, options [nop,nop,TS val 248300786 ecr 3965422857], length 78
10:51:27.801751 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [.], ack 79, win 509, options [nop,nop,TS val 3965422858 ecr 248300786], length 0
10:51:27.802022 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [P.], seq 1:4097, ack 79, win 509, options [nop,nop,TS val 3965422858 ecr 248300786], length 4096
10:51:27.802059 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 4097, win 491, options [nop,nop,TS val 248300786 ecr 3965422858], length 0
10:51:27.802120 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [P.], seq 4097:5396, ack 79, win 509, options [nop,nop,TS val 3965422858 ecr 248300786], length 1299
10:51:27.802135 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 5396, win 502, options [nop,nop,TS val 248300786 ecr 3965422858], length 0
10:51:27.803484 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [F.], seq 79, ack 5396, win 512, options [nop,nop,TS val 248300787 ecr 3965422858], length 0
10:51:27.803567 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [F.], seq 5396, ack 80, win 509, options [nop,nop,TS val 3965422859 ecr 248300787], length 0
10:51:27.803597 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 5397, win 512, options [nop,nop,TS val 248300787 ecr 3965422859], length 0
10:53:08.523431 IP 172.26.64.6.53042 > 95.216.165.210.27372: Flags [.], ack 2169295212, win 501, options [nop,nop,TS val 735542538 ecr 4133067854], length 0
10:53:08.523551 IP 95.216.165.210.27372 > 172.26.64.6.53042: Flags [.], ack 1, win 509, options [nop,nop,TS val 4133379150 ecr 735231242], length 0
10:53:08.523554 IP 95.216.165.210.27372 > 172.26.64.6.53042: Flags [.], ack 1, win 509, options [nop,nop,TS val 4133379150 ecr 735231242], length 0
10:53:08.523562 IP 172.26.64.6.53042 > 95.216.165.210.27372: Flags [.], ack 1, win 501, options [nop,nop,TS val 735542538 ecr 4133379150], length 0

To summarize the output, we can see that the log is showing a TCP connection between 172.26.64.1 (source) and 172.26.64.6 (destination), specifically on port 7000. 172.26.64.1 happens to be the gateway for nomad subnet.

Summary#

Hope this post clarified some networking internals and behind the scenes magic when using Nomad bridge networking. Refer to my other post - Nomad networking explained for a practical breakdown of all the different ways to expose and connect applications in a Nomad cluster.

Fin!

]]> Karan Sharma https://www.divyamohan.com/blog-bts-community-event-part-2/ https://www.divyamohan.com/blog-bts-community-event-part-2/ Fri, 07 Jul 2023 08:05:48 GMT

In the first part of the series, I discussed our why behind organizing a community-driven event that was also the very first edition of Kubernetes Community Days in Mumbai. In this post, we'll dive into the juicy bits of how we went about setting the stage (very literally) for the event.

Our partners

While curating a community event, partnering with likeminded folks is EXTREMELY important - whether it be sponsors, vendors, community collaborators, volunteers, or even organizers. Lack of a shared vision impedes the way you go about planning and executing the event. As organisers, this is a lesson that we were lucky to have learned fairly early on in the process.

I will not delve into the details of how we faced issues on this front from certain actors in the ecosystem.

However, what really helped us despite all these setbacks was dividing & delegating responsibilities amongst the organizing team. This also helped us to play to our strengths & do the work we were each taking responsibility for, well. Since we all were working day jobs and had other external commitments that we were juggling along with organizing the event, staying on 24*7 would result in us spiralling into burnout. That situation wouldn't have been ideal from a personal or professional standpoint and is something I have talked about in a previous KubeCon session.

Our schedule

Speaking of sessions, as co-organizers our top priority was to deliver as much value as possible within 9 hours. This is reflected well in the schedule we created, with a good mix of different formats.

But getting to that point from scratch wasn't the easiest ride. Infact, I'll be the first one to admit this - curating a schedule is the most difficult part of putting together an event.

We received over 150 proposals for a 9-hour event. Yep, you read that right. 150 proposals! 🤯

While deliberating on what would and wouldn't make the cut, over and above the diversity in content we also had to ensure that we had diversity in representation at the event - in terms of

1. speaker gender,
2. professional experience,
3. speaking experience etc.

Being a female in tech myself, this is something I am very keen on ensuring for really selfish purposes - that of having other women & non-males in the room and in the speaker slots.

Unfortunately, despite some of the really cool proposals that were sent our way, we were able to select only 20, i.e. 13.33% of the total CfPs we received, keeping all of the aforementioned criteria in mind.

Of course, the above statistic does not include the slots we reserved for sponsors which formed ~26% of our curated schedule.

Would've, could've, should've...

Looking back, there were several things, we could have done better. In this very context, there's a thread I penned yesterday with some thoughts about multiple conferences within the same region. You can check it out here.

But that is a discussion to be had among a larger audience. However, being a perfectionist (I have been working on this post for 1.5 weeks before hitting the publish button today), it was extremely difficult for me to not have things go "per plan" on the day of and leading up to the event. Whether it be last minute glitches despite testing everything in advance or unforeseen delays, I probably should have accepted my fate (and Murphy's law).

This is why, as retrospective actionable measures, not only will we be publishing an event transparency report with relevant statistics but also will be seeking feedback from all attendees regarding their event experience.

Not only will this help us improve the next conference experience for everyone but it also gives us the opportunity to learn from the community what they'd like to see more of.

So I'll cap this post off with a request, as co-organizers, we look forward to seeing your responses on the feedback survey that we'll be sending out shortly. Please consider sparing a couple of minutes to respond to the email that you receive so that we can ensure y'all have an improved KCD in 2024 to look forward to!

This is the second in a series of posts around organizing KCD Mumbai. Read the first part here.

]]> Divya Mohan Open Source KCD Mumbai Community Blog https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-packages/index.html https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-packages/index.html Thu, 06 Jul 2023 00:00:00 GMT Aditya Athalye programming emacs howto recurse_center https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-init-begins/index.html https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy-init-begins/index.html Tue, 04 Jul 2023 00:00:00 GMT Aditya Athalye programming emacs howto recurse_center https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy/index.html https://www.evalapply.org/posts/emerging-from-dotemacs-bankruptcy/index.html Thu, 29 Jun 2023 00:00:00 GMT Aditya Athalye programming emacs howto recurse_center https://ravidwivedi.in/posts/what-do-free-bus-rides-for-women-do/ https://ravidwivedi.in/posts/what-do-free-bus-rides-for-women-do/ Mon, 26 Jun 2023 19:30:50 GMT Recently, I read news about Karnataka state government announcing free bus rides for women in the state. It was in the Indian National Congress’ pre-election guarantee, so no big surprise on the announcement. I wondered what changes this announcement makes in terms of people’s lives, even when the Delhi government announced such a scheme earlier. I thought such a decision is far from attaining any real goals of women safety and is only an easy way for politicians to earn some goodwill in the name of gender equality, rather than doing any long term work for gender equality. Earlier when I saw advertisements of providing free bus tickets by the Delhi government, they never mentioned any claims on what this move is intended for. One thing I could infer was that greater number of women going out may impact safety, but I was overall not sure on what this policy achieves.

I got a new perspective on the issue when, a couple of days ago, I read an article by The Guardian on the same topic. Then I found another article with similar arguments in the Indian Express. Later I read a report in the Times of India on how free bus ride scheme helped women in Chennai. These writeups shed light on the benefits of free bus rides to women which I could not think of myself (shows my ignorance:( ).

You can click on the links of the articles and read them there. But I will summarize here: In India, many women rely on family males for money (which is also used as a form of control) and so they have to rely on their savings to travel. I also had misconception, due to the scheme’s counterpart in Delhi, that the fares of city buses are not much to make a real difference. But in the Karnataka case they are. I know from my experience that Karnataka’s capital city Bengaluru has expensive public transport compared to other Indian cities. For example, bus fare from Bengaluru Airport to Satellite Bus station, which is also in Bengaluru, was 246 Indian Rupees six years ago, and I have never seen such a high price in a city bus run by government within any other Indian city. Also, due to scheme being available in the whole state, which I somehow overlooked, women will be able to get free rides for which they would have to pay huge amount otherwise. This will boost their savings, ease travel for women, increase the number of women in public transport leading to even more women travelling in buses. It is also expected to increase women participation in labour which matters in a country where women labour participation is very low. Obviously, such a scheme also needs to be evaluated in terms of economic terms and bigger picture of state finances, but the point is that the scheme has a lot of potential and upsides, where I struggled to find any.

]]> Ravi Dwivedi https://www.divyamohan.com/blog-bts-community-event/ https://www.divyamohan.com/blog-bts-community-event/ Thu, 22 Jun 2023 06:14:32 GMT I'll be the first one to admit - I had absolutely no plans of writing anything about organizing community-driven events. After all, organizing one (and helping co-organize another, as a volunteer) didn't make me an expert.

Additionally, as far as I was concerned, there was nothing new I could add that my co-organizers, Saiyam, Rohit, Shivay, and I hadn't already said at the event or in our posts on social media

However, as I was sifting through the love we received on social media over the past week, I saw this tweet by Ramiro Berrelleza, founder of Okteto.

While we've all given thanks to the people who partnered with us and posted really nice photos, nobody had ever shone light on the blood, sweat, and tears that went behind putting together the event from scratch. And this was the moment a light bulb went off in my head.

I know, I know, this story is less fascinating than how a certain Mr. Archimedes discovered buoyancy.

But, it honestly is how the post you're reading came to be. I intend for this BTS to be a series because there's way too much content to fit into a single post.

So, let's start with the basics, shall we?

The why

If you've ever heard me talk/present, I normally start by discussing why I chose to speak about the topic at hand. Nothing to do with Simon Sinek's bestselling book, of course. I'm just a rambler & a highly anxious person who LOVES to justify and that is exactly how we'll be starting off this series - by looking at the why behind KCD Mumbai.

So, for anyone who has heard about India or lived in India, you'd know that the hub for everything tech is Bengaluru. It is a lovely city with some very tasty food (Ghee roast & filter coffee at MTR, anybody?) and GREAT weather.

Other emerging tech hubs in India with a lot of startup action & MNC operations are the cities of Pune & Hyderabad.

Contrast these options with Mumbai, which is also a metropolitan city, has a humid climate, is crowded, and is not really renowned for its tech scene; and the  question arises, why Mumbai?

As advocates for cloud native technologies at our day jobs & outside of it, teaming up with KCD Bengaluru or KCD Chennai would have been our best bet. We would not need to compete for company sponsorships in these economic conditions & we'd also probably be able to reach our target audience better since they work in and around these cities.

However, if I could sum up our rationale behind hosting Kubernetes Community Days in Mumbai in a single word, it would be accessibility. As organizers, we wanted to host an event replete with international speakers, sponsors etc. that folks from other parts of the country, especially tier-2 & tier-3 cities, were able to attend.

Now Mumbai might not be endowed with great weather, but one thing that it does boast of is connectivity to almost all other Indian cities - by flight, road, or rail.

By hosting this event in Mumbai, not only did we want to encourage people from other parts of the country to attend such events, but we also hoped to encourage attendees to spin up local chapters in their respective cities by example. Of course, we wouldn't expect you to believe what we say without demonstrating it!

This is not to diss the efforts of any other Kubernetes Community Days being hosted in India. As a matter of fact, we hope that by attending events like these, more folks are inspired to come forth, engage with the cloud native & open source ecosystems, and spin up local tech scenes of their own.

Also, with this event, we wanted to reinstate the message that you've probably heard ever so often from different people, "Open source contributions aren't just about code." While I'm a documentation maintainer for the Kubernetes project and a CNCF ambassador, there were folks in the volunteering & organizing team who chose to (succesfully) engage with and contribute to the ecosystem in ways other than code & documentation.

Once we had narrowed down our "why" , it also helped us find our where and who (more on that later). With this GitHub issue , we took the first step toward formalizing our intent and get the ball rolling for further discussion. In the next post, I intend to delve into the juicy details of the areas we, as organisers, had to focus on to put together this event from scratch - sponsorship,volunteers, swag, et al.

But before I end this post, if you or anyone you know was inspired by KCD Mumbai/any other CNCF event and you'd like to host one locally, please do check out the CNCF Community Groups page. There are a bunch of fantastic resources for you to get started and help with making cloud native (and open source) ubiquitous.

]]> Divya Mohan Open Source KCD Mumbai Community https://shrirangkahale.com/posts/insider-threat/ https://shrirangkahale.com/posts/insider-threat/ Wed, 21 Jun 2023 13:00:50 GMT Shrirang Kahale https://www.evalapply.org/posts/what-have-you-been-curious-about/index.html https://www.evalapply.org/posts/what-have-you-been-curious-about/index.html Wed, 21 Jun 2023 00:00:00 GMT Aditya Athalye meta riff https://www.learningfromdata.zingg.ai/p/building-identity-resolution-on-snowflake https://www.learningfromdata.zingg.ai/p/building-identity-resolution-on-snowflake Mon, 19 Jun 2023 12:32:35 GMT

I have very exciting news to share - we have signed our first customer for Zingg Enterprise Snowflake! It feels magical, almost surreal. I call it magical not because I did not believe it will happen, but because of the pace and ease of closing the deal. If you have been doing B2B enterprise sales, you probably know what I mean :-)

Zingg open source has always supported Snowflake as a data source using Apache Spark as the connector. As we talked with our users, we realized that many of them had only SQL or warehouse workloads, and we could really simplify their stack by using Snowflake directly. Over the last year, we watched Snowpark evolve and did multiple experiments to see how viable it would be to build Zingg over Snowpark. This post will detail some of the things we did and learned as part of the process.

Our first challenge was building our extensive ML pipeline for entity resolution in Snowpark. We have our own blocking algorithm but rely on Spark’s MLLib for classification. Snowpark does not have a built in machine learning library but facilitates ML through third party packages like scikit-learn. This means that Snowpark Dataframes can not be passed directly for training models, but a conversion to Pandas Dataframe becomes necessary. This has two issues. Firstly, this limits the size of data one can train on. Fortunately for us, as we use active learning to build training sets efficiently, our training data size is quite small. The second issue was the real one for us. Our code is in Java and Scala so introducing a Python flow for training and prediction became challenging. Zingg does substantial preprocessing and feature engineering enriching the input data with various signals before training and prediction. Using Python meant our in-memory enriched Dataframe could not be passed directly for training. Eventually, we decided to persist the features into a temporary table, and built a stored procedure in Python to read it to train a classifier. We persist the classifier into a Snowflake table. and load it in a named Python user defined function, which is invoked from our codebase during inference. Thankfully, a named user defined function in one language can be easily invoked from another, which is the strategy we have used extensively in our code.

Snowpark also misses a graph library, like GraphX or Graphframes in Apache Spark. Luckily the Snowflake team helped us here and provided us with a Scala library for graph processing. We had some work cut out to integrate this into our code, especially since the Snowpark Scala Dataframes are not interchangeable with the Snowpark Java ones.

Building a cross-domain ML product introduces a great deal of generalization in the flow. We do not know beforehand the shape of the input dataset - it could be product matching, it could be customer matching, or it could be a B2B account with domain names and other details. Many row-wise operations become tricky without a map equivalent in Snowpark. Hence we built our own version of map, passing all the columns into a VARIANT type array and then getting individual values as necessary. This was a bit of a tough problem, and we were pretty proud when we could get it to work.

There were some other minor hiccups like temporary udfs not getting cleaned up, no way to access data from dataframes using column names but having to pass column index, etc. I am sure these will get sorted out as the Snowpark API matures further.

One of our biggest concerns was that our codebase assumed Apache Spark as the defacto processing engine, meaning that the dependencies were all over the code. Apache Spark is an integral part of Zingg, and is intricately weaved into all Zingg operations. So our first thought was to have two code bases, one for Apache Spark and then the same logic ported for Snowpark. All functionality would exist independently in each. There would be some common code like reading user configuration that would be shared between the two. As we were planning the Snowflake version of the product to be closed source, the code would anyways have to be in a different repository than the open source ones. Hence, why not? This would also let us build the product in less time, as the open source is extensively validated and already in production. Ship faster, they say! We spent a day or two doing global search and replace operations on dependencies and imports, to see how this would shape out and realized soon enough that this was going to be a mess. If we had to fix a bug, let alone add a new feature, we would have to do it in two places. Clearly, this was going to be a recipe for a lot of overwork and technical debt and I am glad we abandoned this soon enough.

The next approach was to build up generic classes and then templatize them with Spark or Snowpark equivalents. All the logic remains centrally located in one place, and the processing engine equivalents have their own extensions. We can easily add another processing engine if a new Dataframe API comes up. This meant touching every single bit of our code, which was going to be a lot of work on both the development and the testing fronts. Not a decision we wanted to jump into, but there wasn’t a lot of choice here. I have to let you in a secret here. Besides the user discussions, we had another motive for the Snowflake product. We wanted to enter the Snowflake challenge. Some cash and publicity can never hurt a frugal startup! We wanted to apply with a running application, which left us with roughly 1.5 months to get everything running, and that is the deadline we set for ourselves at the beginning of the year. It was a lot of work, and more work, and more work. It was tiring, but it was fun. We finished a week before our self-induced deadline, without any shortcuts or hacks :-)

Snowpark has been recently open-sourced, so it is going to be even more fun building around it. Zingg is surely one of the more complex applications being built around it, and we can’t wait to add more stuff to our identity resolution product and push Snowpark further.

In the end, we did not win the challenge, but had some nice interactions with the Snowflake team and have learned a lot of stuff that will help us as we now put Zingg Identity Resolution on Snowflake into production.

We are really excited about how this journey is shaping up!

]]> Sonal Goyal https://www.evalapply.org/posts/software-demos/index.html https://www.evalapply.org/posts/software-demos/index.html Sun, 04 Jun 2023 00:00:00 GMT Aditya Athalye programming culture whyto software_design meta https://www.prashanthudupa.com/learning-backwards/ https://www.prashanthudupa.com/learning-backwards/ Wed, 31 May 2023 14:47:35 GMT Prashanth Udupa Philosophy Unschooling https://nadh.in/blog/this-time-it-feels-different/ https://nadh.in/blog/this-time-it-feels-different/ Sat, 13 May 2023 00:00:00 GMT In an earlier post (2021) , I argued that much of the “powered by AI / ML” labelling and marketing out there was bogus and disingenuous. That AI / ML technologies were getting commoditised to the point of being as simple as pip install, where most organisations would not need to do any serious R&D to be able to use these technologies, enough to warrant the claim “Powered by AI / ML”. Excerpt from the post:

]]> Kailash Nadh https://shrirangkahale.com/posts/learning-journey/ https://shrirangkahale.com/posts/learning-journey/ Sat, 06 May 2023 18:29:57 GMT Shrirang Kahale https://shrirangkahale.com/about/ https://shrirangkahale.com/about/ Sat, 06 May 2023 00:00:00 GMT Shrirang Kahale https://shrirangkahale.com/contact/ https://shrirangkahale.com/contact/ Sat, 06 May 2023 00:00:00 GMT Shrirang Kahale https://www.prashanthudupa.com/i-can-afford-it-but-he-chooses-to-do-jugaad/ https://www.prashanthudupa.com/i-can-afford-it-but-he-chooses-to-do-jugaad/ Wed, 03 May 2023 08:11:19 GMT Prashanth Udupa Moments Unschooling https://shrirangkahale.com/posts/airtel_vi_routing/ https://shrirangkahale.com/posts/airtel_vi_routing/ Mon, 01 May 2023 09:53:15 GMT Shrirang Kahale https://mrkaran.dev/posts/analyzing-credit-card-transactions/ https://mrkaran.dev/posts/analyzing-credit-card-transactions/ Sun, 30 Apr 2023 07:08:02 GMT 0] credit_entries = df[df["Amount (INR)"] < 0] # Count the total number of debit and credit entries total_debit_entries = len(debit_entries) total_credit_entries = len(credit_entries) # Calculate the total amount of debits and credits total_debit_amount = debit_entries["Amount (INR)"].sum() total_credit_amount = credit_entries["Amount (INR)"].sum() # Find the vendor with the most spending most_spent_vendor = debit_entries.groupby("Details")["Amount (INR)"].sum().idxmin() # Calculate the total amount spent on SWIGGY swiggy_spending = debit_entries[debit_entries["Details"].str.contains("SWIGGY", case=False)]["Amount (INR)"].sum() # Calculate the total amount spent on AMAZON RECHARGES amazon_recharges_spending = debit_entries[debit_entries["Details"].str.contains("AMAZON RECHARGES", case=False)]["Amount (INR)"].sum() # Print the results print(f"Total debit entries: {total_debit_entries}") print(f"Total credit entries: {total_credit_entries}") print(f"Total debit amount: {total_debit_amount}") print(f"Total credit amount: {total_credit_amount}") print(f"Vendor with the most spending: {most_spent_vendor}") print(f"Total amount spent on SWIGGY: {swiggy_spending}") At this point, I know that this will fail because we’ve not cleaned up the data. The column Amount (INR) needs cleaning up. I prompted ChatGPT to write a function to clean this column: The amount column as `21,620.54 Dr.` to represent that Rs 21,620.54 is debited. And `73,500.00 Cr.` to represent that Rs 73,500.00 is credited. Can you write a function to apply this transformation on the `Amount (INR)` column? It responded with: # Clean up the data def clean_amount(value): if ' Cr.' in value: value = value.replace(' Cr.', '') return -float(value.replace(',', '')) elif ' Dr.' in value: value = value.replace(' Dr.', '') return float(value.replace(',', '')) df["Amount (INR)"] = df["Amount (INR)"].apply(clean_amount) # Now proceed with the analysis code Perfect! After transforming the amounts, I ran the analysis code: The next prompt I gave was to analyse the spending in various categories. In the above CSV file can you guess the categories of each merchant based on the `Details`? And write the code for grouping the spends on various categories. If you're unable to figure out the category for the transaction, you can use a "Misc" column to bucket them. It actually did an okayish job at this and ignored a lot of vendors which I think it could have guessed easily: def categorize(details): details = details.lower() if "swiggy" in details: return "Food Delivery" elif "amazon recharges" in details: return "Recharge" elif "uber" in details or "ola" in details: return "Cab" elif "zomato" in details: return "Food Delivery" elif "netflix" in details or "prime video" in details: return "Streaming" else: return "Misc" # Create a new column called "Category" by applying the categorize function to the "Details" column df["Category"] = df["Details"].apply(categorize) # Group the transactions by category and calculate the total amount spent in each category grouped = df[df["Amount (INR)"] > 0].groupby("Category")["Amount (INR)"].sum() print(grouped) I decided to give some manual inputs to it to refine the function. I can help you with categorising it further. `Blink` - Grocery `Indigo` - Travel `PEGS` - Alcohol `Nike` - Shopping Also note that the cateorgy name matching should be case-insensitive. And it modified the Python snippet to add these rules. Mixed reactions looking at this. Happy that I could practically get exactly the result I had in mind in just 10 minutes without writing any code. Sad because damn I need to limit those empty calories from next month (famous last words). Next, I wanted to see if my spends on weekends are higher or not. (I don’t expect them to be, but you never know). Prompt: Can you also write the code to figure how much spends I've done grouped by the day of week? Response: # Convert the 'Transaction Date' column to a datetime format df['Transaction Date'] = pd.to_datetime(df['Transaction Date']) # Create a new column called "Day of Week" containing the day of the week for each transaction df['Day of Week'] = df['Transaction Date'].dt.day_name() # Group the transactions by the "Day of Week" and calculate the total amount spent on each day of the week grouped_by_day = df[df["Amount (INR)"] > 0].groupby('Day of Week')['Amount (INR)'].sum().abs() # Sort the grouped data by the total amount spent in descending order sorted_grouped_by_day = grouped_by_day.sort_values(ascending=False) print(sorted_grouped_by_day) It was a fun 10-15 min exercise to figure out my spending habits based on the last month’s statement. I intend to do this for the next couple of months and then it would make sense to write more queries which would show trend-lines of spends in various categories over time. Honestly, I just loved how ChatGPT made this task so seemingly simple. It’s not that I can’t myself write the code for these kind of simple analysis. It’s the sheer power at your hand to go from ideation phase to answer within seconds. And I think that’s why I love it so much. I didn’t have to go through Pandas docs (because I don’t use that in my day job so it’s quite normal to not know various syntax/functions that I could use) and I’d grok through different StackOverflow questions to achieve what I wanted to. And maybe imagining all this resistance on a Sunday morning would have meant that I never got to write the script in the first place. Fin!]]> You know those budget freaks? People who log and categorise every Rupee they’ve spent over the month? The financially sane people? I am definitely not one and I suck at it.

I moved cities a couple of months back and had some big ticket spends off late, mostly financed by credit card. I wanted an easy way to list down where all I’ve spent most money and spot some recurring expenses so I can be better prepared for them from next month. I’ve found that broadly keeping an idea of things where you spend money works for me (v/s the two extremes - completely blind or logging every small transactions). Of course I know people who make budgeting a habit but I only wish I was consistent enough to do that.

Anyway, I downloaded the statement in CSV format from my bank.

Initially I thought I’d use some simple Excel to make sense of this but I realised how bad my Excel skills really are. I got an idea to dump the CSV file to ChatGPT (yay privacy) and ask questions. It kinda sucked at it and gave wrong answers for a lot of questions and also started to hallucinate data which wasn’t even present in the CSV.

The next most obvious step would be to write a simple script and parse it. I wanted to experiment if ChatGPT could do this entire exercise of writing the script and the relevant code for the analysis I wanted to perform.

Here’s the initial prompt I gave:

I am pasting a CSV file which also includes the first row as header. Write Python code to parse this:

,,Transaction Date,Details,,,Amount (INR),,,Reference Number,,
,,27/04/2023,"Blinkit, BANGALORE, IN",,,290.00 Dr.,,,7477468633,,
... other data ...

It returned the following code:

import pandas as pd
from io import StringIO

csv_data = """\
,,Transaction Date,Details,,,Amount (INR),,,Reference Number,,
,,27/04/2023,"Blinkit, BANGALORE, IN",,,290.00 Dr.,,,747746xxxx,,
# ... (the rest of your CSV data)
,,03/04/2023,"DIGITAL LOUNGE, GURGAON, IN",,,799.00 Dr.,,,738201xxx,,"""

# Read the CSV data into a pandas DataFrame
data = StringIO(csv_data)
df = pd.read_csv(data, header=1)

# Clean up the DataFrame (remove unnecessary columns)
df = df.dropna(axis=1, how="all")

# Rename columns
df.columns = [
    "Transaction Date",
    "Details",
    "Amount (INR)",
    "Reference Number"
]

print(df)

Looking at this, I was a bit impressed as it figured the CSV contains some useless empty columns and it removed (without me giving any information about it).

I also asked it to modify the code to read the file locally from disk and it swapped StringIO with the path to CSV file:

# Provide the file path to your local CSV file
file_path = "your_file.csv"

# Read the CSV data into a pandas DataFrame
df = pd.read_csv(file_path, header=1)

Next, I prompted to do some analysis on it:

Write some more code to do the analysis to answer these questions:

- How many total debit and credit entries
- What is the total amount of debits and total amount of credits
- Which vendor have I spent the most?
- How much have I spent on `SWIGGY` ?

It returned some one-liners to answer each question:

# Filter debit and credit entries
debit_entries = df[df["Amount (INR)"] > 0]
credit_entries = df[df["Amount (INR)"] < 0]

# Count the total number of debit and credit entries
total_debit_entries = len(debit_entries)
total_credit_entries = len(credit_entries)

# Calculate the total amount of debits and credits
total_debit_amount = debit_entries["Amount (INR)"].sum()
total_credit_amount = credit_entries["Amount (INR)"].sum()

# Find the vendor with the most spending
most_spent_vendor = debit_entries.groupby("Details")["Amount (INR)"].sum().idxmin()

# Calculate the total amount spent on SWIGGY
swiggy_spending = debit_entries[debit_entries["Details"].str.contains("SWIGGY", case=False)]["Amount (INR)"].sum()

# Calculate the total amount spent on AMAZON RECHARGES
amazon_recharges_spending = debit_entries[debit_entries["Details"].str.contains("AMAZON RECHARGES", case=False)]["Amount (INR)"].sum()

# Print the results
print(f"Total debit entries: {total_debit_entries}")
print(f"Total credit entries: {total_credit_entries}")

print(f"Total debit amount: {total_debit_amount}")
print(f"Total credit amount: {total_credit_amount}")

print(f"Vendor with the most spending: {most_spent_vendor}")

print(f"Total amount spent on SWIGGY: {swiggy_spending}")

At this point, I know that this will fail because we’ve not cleaned up the data. The column Amount (INR) needs cleaning up. I prompted ChatGPT to write a function to clean this column:

The amount column as `21,620.54 Dr.` to represent that Rs 21,620.54 is debited. And `73,500.00 Cr.` to represent that Rs 73,500.00 is credited. Can you write a function to apply this transformation on the `Amount (INR)` column?

It responded with:

# Clean up the data
def clean_amount(value):
    if ' Cr.' in value:
        value = value.replace(' Cr.', '')
        return -float(value.replace(',', ''))
    elif ' Dr.' in value:
        value = value.replace(' Dr.', '')
        return float(value.replace(',', ''))

df["Amount (INR)"] = df["Amount (INR)"].apply(clean_amount)

# Now proceed with the analysis code

Perfect! After transforming the amounts, I ran the analysis code:

The next prompt I gave was to analyse the spending in various categories.

In the above CSV file can you guess the categories of each merchant based on the `Details`? And write the code for grouping the spends on various categories. If you're unable to figure out the category for the transaction, you can use a "Misc" column to bucket them.

It actually did an okayish job at this and ignored a lot of vendors which I think it could have guessed easily:

def categorize(details):
    details = details.lower()
    if "swiggy" in details:
        return "Food Delivery"
    elif "amazon recharges" in details:
        return "Recharge"
    elif "uber" in details or "ola" in details:
        return "Cab"
    elif "zomato" in details:
        return "Food Delivery"
    elif "netflix" in details or "prime video" in details:
        return "Streaming"
    else:
        return "Misc"

# Create a new column called "Category" by applying the categorize function to the "Details" column
df["Category"] = df["Details"].apply(categorize)

# Group the transactions by category and calculate the total amount spent in each category
grouped = df[df["Amount (INR)"] > 0].groupby("Category")["Amount (INR)"].sum()

print(grouped)

I decided to give some manual inputs to it to refine the function.

I can help you with categorising it further.
`Blink` -  Grocery
`Indigo` - Travel
`PEGS` - Alcohol
`Nike` - Shopping

Also note that the cateorgy name matching should be case-insensitive.

And it modified the Python snippet to add these rules.

Mixed reactions looking at this. Happy that I could practically get exactly the result I had in mind in just 10 minutes without writing any code. Sad because damn I need to limit those empty calories from next month (famous last words).

Next, I wanted to see if my spends on weekends are higher or not. (I don’t expect them to be, but you never know).

Prompt:

Can you also write the code to figure how much spends I've done grouped by the day of week?

Response:

# Convert the 'Transaction Date' column to a datetime format
df['Transaction Date'] = pd.to_datetime(df['Transaction Date'])

# Create a new column called "Day of Week" containing the day of the week for each transaction
df['Day of Week'] = df['Transaction Date'].dt.day_name()

# Group the transactions by the "Day of Week" and calculate the total amount spent on each day of the week
grouped_by_day = df[df["Amount (INR)"] > 0].groupby('Day of Week')['Amount (INR)'].sum().abs()

# Sort the grouped data by the total amount spent in descending order
sorted_grouped_by_day = grouped_by_day.sort_values(ascending=False)

print(sorted_grouped_by_day)

---

It was a fun 10-15 min exercise to figure out my spending habits based on the last month’s statement. I intend to do this for the next couple of months and then it would make sense to write more queries which would show trend-lines of spends in various categories over time.

Honestly, I just loved how ChatGPT made this task so seemingly simple. It’s not that I can’t myself write the code for these kind of simple analysis. It’s the sheer power at your hand to go from ideation phase to answer within seconds. And I think that’s why I love it so much. I didn’t have to go through Pandas docs (because I don’t use that in my day job so it’s quite normal to not know various syntax/functions that I could use) and I’d grok through different StackOverflow questions to achieve what I wanted to. And maybe imagining all this resistance on a Sunday morning would have meant that I never got to write the script in the first place.

Fin!

]]> Karan Sharma https://www.prashanthudupa.com/you-should-know-this-stuff-by-now/ https://www.prashanthudupa.com/you-should-know-this-stuff-by-now/ Mon, 24 Apr 2023 08:06:00 GMT Prashanth Udupa Moments Unschooling https://www.learningfromdata.zingg.ai/p/measure-what-matters-our-simple-data https://www.learningfromdata.zingg.ai/p/measure-what-matters-our-simple-data Tue, 18 Apr 2023 16:05:53 GMT

What good is a song composed but not hummed by the audience? The musician may love it as their best song, but it will soon fade away from the memory of its listeners. While the pleasure of building and innovating is its own reward, creators need external validation to generate the ammunition to make even bolder bets.

As people who pour our hearts and souls into building Zingg, we too take a pause at times to reflect if we are indeed creating value for our users. There are two main things we are super curious to learn at this stage.

Thanks for reading Sonal’s Newsletter! Subscribe for free to receive new posts and support my work.

1. To learn how people are discovering us

2. To understand how they are using Zingg

Signals for these come from multiple sources. The important ones are direct user and customer communication, Slack, website traffic, product downloads and product usage statistics.

Direct interaction with users and customers is the richest form of learning. Many of our open source users join our slack and chat with us and we learn about the problems they are solving with Zingg. Some of them write to us on email, or DM us on Slack or LinkedIn, sharing what they are building with and around Zingg. They want to check the best way to use Zingg for their scenario. We ask them about their stack and use cases, and how we can help them better. Learning more about our users guides us in shaping the roadmap for Zingg. Zingg’s docker and the Zingg Python API are a direct result of these interactions. We built them because we realized that they will help users generate identity resolution results faster and will enable them to use Zingg as a first class citizen in their transformation pipelines.

While interacting directly, we often ask users where they learnt about Zingg. This is augmented with the signals from traffic information from Google Analytics. Github also provides repository traffic analysis of the previous fortnight.

Often, the discovery is word of mouth. For us, having a great product and providing an even better support is the best form of engaging people. It suits us as developers, it helps our users drive value faster and it builds a strong relationship with the user. A check at the testimonials on the Zingg website reveals we are doing a nice job there.

And we should still aim to get better here. Like identity resolution, support is never going to be perfect, so we always have things to improve upon.

Many times the discovery is through the tutorials and guides we have written. Or talks and events in select data communities. Instead of a lot of content, we have focused on deep and long articles that help the user understand the problem of identity and entity resolution, why it is needed, the challenges in resolving entities and how Zingg can be run on different platforms like Databricks and Snowflake. Early adopters of a product know the problem and are looking for a solution. Hence, we owe it to them to make using Zingg easy so that they can focus on solving the bigger puzzle - Customer 360, Segmentation, Fraud, Risk, Compliance etc. Discoverability is the side benefit of our content, helping our users adopt Zingg is the main goal.

Broadly, these data points are sufficient for us to understand and improve discoverability at this stage. Besides this, we track open source adoption and product usage anonymously. We gather information about

1. Github downloads of our releases

2. Docker pulls

3. PyPi installs

4. Slack users

5. Documentation views

Together, the above give us a fair idea of our adoption and how we are growing. For now, we get these values manually every couple of weeks and track them in an excel file, charting out a time series to understand our growth rate. We do have a column for github stars which we populate but dont think much about as we have richer data to get signals from.

Coming to product analytics, the aim here is to learn the different aspects of how people are using Zingg-

1. Which platform do they run Zingg on?

2. Is the product scaling? How many records do they match?

3. How much time does it take to run?

4. Do they run Zingg multiple times or is it a one time deduplication?

For any product analytics to function, we ought to have a way to identify unique installations and trigger usage events from the installation to a place where we can analyze them. Our asks are fairly simple questions, but the novel nature of our product as well as deployment makes it tricky to get the answers.

Often, Zingg is deployed on cloud resources like Databricks or EMR or on private enterprise networks which are blocked from sending information outside the VPN. So if a user downloads and uses Zingg under such environments, we have no way of knowing how they are using Zingg.

Another issue is that it is impossible to identify installs through cloud systems, as virtual machines change with every run. Even when we don’t want to identify the user but only the install, there is no way to track a unique installation by writing a random uuid on disk. This problem exists on our docker environment as well.

Some of the features we are building now will allow users to trigger cloud jobs from their machines, so we will think about the install identification then. Till then, we already have the download stats, so thats a good enough proxy for us. Instead, we gather usage data. We respect user privacy, so we limit data collection to reflect product usage. We also provide a flag to turn data collection off.

We use Google Analytics as our event server and post custom events to it from the code every time Zingg is run. These events are sent to our GA account, which has been configured to pipe these events into BigQuery. For every run of Zingg, we collect the data and output formats, time it took to run, number of records and a few other stats. Every few weeks, a handful of SQL queries help us understand the data volumes we are processing, the number of daily jobs run and their trends, the time taken to complete each run etc. Mostly, it is a brief glance on the numbers to make sure we are on the right track.

Many times, the usage data has an unheard story to tell. We discovered that there were a good amount of users using Zingg to resolve entities on Snowflake. A Spark deployment is not ideal for them, as they are not using it anywhere in their stack. Could we leverage Snowpark and build a lighter, tighter product for such users? Seems we can, and I have a lot of exciting stories to tell about that effort and where it is leading us!

Coming back to our simplistic data stack.

Sometimes we do detective work ;-) If we see events of the same format and number of fields triggered at regular time intervals through the day, we treat it as Zingg running in production. Once in a while, it is fun to look into individual events and extrapolate the usage to different scenarios.

Churning out aggregate stats is a revelation too - like Zingg has resolved at least 2 billion records in the last two months. This is a huge number, and it has grown rapidly over the last few months.

When I say “at least”, it is with the awareness that we are not collecting all the data points. We have talked with many users who are running production loads but they do not show up in our telemetry. So our analysis tends to be conservative. Which is absolutely fine.

Overall, the aim of this stack is to help us get a sense of how we are doing, what we can do to help our users and what more we should build. We are on free plans on Slack as well as GA. The frugality is in the cost, and also in the implementation. As a small team, we needed something that is easy to setup and will answer our basic questions without much effort. Also, the trend in the data is more important than the individual data points at this stage.

We may miss some dots, but the path is clear. Which is what matters!

Thanks for reading Sonal’s Newsletter! Subscribe for free to receive new posts and support my work.

]]> Sonal Goyal https://aryak.me/blog/02-mdad.html https://aryak.me/blog/02-mdad.html Mon, 17 Apr 2023 12:39:46 GMT Yesterday, we completed Project Segfault’s migration from matrix.org’s official docker image for synapse to matrix-docker-ansible-deploy.

This was because of how much of a pain it is to setup workers, especially with docker. The docs aren’t great about it either..

For these reasons, we turned to matrix-docker-ansible-deploy.

The first issue we encountered was how spread out the docs were, though very precise and well-explained.

Once we cloned the repo, we first had to setup the inventory hosts file.

Since we cloned the repo to the DockerVM itself, we had a weird solution for this.

[matrix_servers]
matrix.projectsegfau.lt ansible_host=localhost ansible_ssh_user=root

After this, we had to add the pubkey of the VM to its own authorized_keys. Wacky :P

With that sorted, we had to start configuring.

Firstly, we had to prevent it from installing docker.

This is important since (re)installing docker will break a lot, especially for our pre-existing services.

matrix_playbook_docker_installation_enabled: false

After that, we had to add our old secret keys back to the config file so that it won’t break federation:

matrix_synapse_macaroon_secret_key: "xxx"
matrix_synapse_registration_shared_secret: "xxx"
matrix_synapse_form_secret: "xxx"

The signing key had to be re-added as well, but later after the setup was complete.

After that, we turned to the thing we migrated for, synapse workers:

Since the generic and federation_sender workers have to process a lot of data, we made 4 of each (totally didn’t copy the number from envs.net :P).

matrix_synapse_workers_enabled: true
matrix_synapse_workers_preset: one-of-each
matrix_synapse_workers_federation_sender_count: 4
matrix_synapse_workers_generic_worker_count: 4

Another important thing we had to take into consideration was postgres. We ran postgres on a separate VM and connected to the database on it.

matrix_synapse_database_host: "192.168.5.4"
matrix_synapse_database_user: "synapse"
matrix_synapse_database_password: "xxx"
matrix_synapse_database_database: "synapse"
devture_postgres_enabled: false

After the database, we had to set up registration/login stuff.

A weird thing I noticed about matrix-docker-ansible-deploy’s email configuration is that it uses its own relay, above our mail credentials.

matrix_mailer_sender_address: "matrix@projectsegfau.lt"
matrix_mailer_relay_use: true
matrix_mailer_relay_host_name: "mail.projectsegfau.lt"
matrix_mailer_relay_host_port: 587
matrix_mailer_relay_auth: true
matrix_mailer_relay_auth_username: "matrix@projectsegfau.lt"
matrix_mailer_relay_auth_password: "xxx"
matrix_synapse_registrations_require_3pid: [ email ]
matrix_synapse_enable_registration: true
matrix_synapse_configuration_extension_yaml: |
  oidc_providers:
    - idp_id: authentik
      idp_name: "authentik"
      idp_icon: "mxc://envs.net/429bd4b307d32b919a94823f03acc7c24a7da61f"
      discover: true
      issuer: "https://auth.p.projectsegfau.lt/application/o/matrix/"
      client_id: "xxx"
      client_secret: "xxx"
      scopes:
        - "openid"
        - "profile"
        - "email"
      user_mapping_provider:
        config:
          localpart_template: "{% raw %}{{ user.preferred_username }}{% endraw %}"
          display_name_template: "{% raw%}{{ user.name }}{% endraw %}"
          email_template: "{% raw %}{{ user.email }}{% endraw %}"

Past this, we also had to port the small configurations we had in our old homeserver.yaml to the ansible format.

Since most of these weren’t documented very well, we had to make heavy use of the defaults file.

matrix_synapse_auto_join_rooms: [ '#project-segfault:projectsegfau.lt', '#support:projectsegfau.lt', '#general:projectsegfau.lt', '#announcements:projectsegfau.lt' ]
matrix_synapse_max_upload_size_mb: 700
matrix_synapse_allow_public_rooms_without_auth: true
matrix_synapse_allow_public_rooms_over_federation: true
matrix_synapse_email_client_base_url: "https://matrix.to"
matrix_synapse_email_invite_client_location: "https://chat.projectsegfau.lt"
matrix_synapse_turn_uris: ["turn:turn.projectsegfau.lt?transport=udp", "turn:turn.projectsegfau.lt?transport=tcp"]
matrix_synapse_turn_shared_secret: "xxx"
matrix_synapse_turn_allow_guests: true
matrix_coturn_enabled: false
matrix_client_element_enabled: false

At this point we realized that we need to do a lot of weirder stuff to get it to work reverse-proxied behind our main caddy instance.

We reverse-proxied the traefik instance behind our caddy instance, as recommended by the documentation with the instructions there:

# Ensure that public urls use https
matrix_playbook_ssl_enabled: true

# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval
devture_traefik_config_entrypoint_web_secure_enabled: false

# If your reverse-proxy runs on another machine, consider using `0.0.0.0:81`, just `81` or `SOME_IP_ADDRESS_OF_THIS_MACHINE:81`
devture_traefik_container_web_host_bind_port: '0.0.0.0:81'

# We bind to `127.0.0.1` by default (see above), so trusting `X-Forwarded-*` headers from
# a reverse-proxy running on the local machine is safe enough.
devture_traefik_config_entrypoint_web_forwardedHeaders_insecure: true
devture_traefik_additional_entrypoints_auto:
  - name: matrix-federation
    port: 8449
    host_bind_port: '0.0.0.0:8449'
    config: {}

After all the configuration was done, we had to run it :P.

Firstly, we had to install ansible and just, and run just roles to initialize all the ansible stuff.

At this point, we shut down our old matrix instance in order to not cause any issues.

Then, we ran ansible-playbook -i inventory/hosts setup.yml --tags=install-all to install all the files but not start the services.

Now came the most time consuming part, importing the old media repo. Considering its size at over 85 gigabytes.

ansible-playbook -i inventory/hosts setup.yml --extra-vars='server_path_media_store=/opt/docker/mtrx/files/media_store' --tags=import-synapse-media-store

This took almost 30 minutes, the majority of the downtime we had..

After this was done, we were able to start the server: ansible-playbook -i inventory/hosts setup.yml --tags=start.

The nginx configuration they recommended in the documentation for our reverse-proxy setup was pretty self-explanatory and easy to convert, but for the fact that till now our matrix instance used normal delegation and did not make use of :8448.

Due to this, we had to waste a lot of time trying to figure out which routes went to which ports. I wish the documentation explained this better..

At the end, this was the caddy configuration we came up with for this:

matrix.projectsegfau.lt {
    reverse_proxy /_matrix/* 192.168.5.2:8449
    reverse_proxy /_matrix/client/* 192.168.5.2:81
    reverse_proxy /_synapse/* 192.168.5.2:81
}

This configuration works right now, though we are still not completely sure if other routes need to go somewhere else.

I do have some gripes with it though, such as the ages it takes for restarts (–tags=setup-build and then –tags=restart for those wondering) and the lack of documentation for what is the recommended upstream delegation configuration.

At the end, matrix-docker-ansible-deploy simplified our config a lot and relieved a lot of maintanence burden we would have had in case we configured it manually and I am thankful for that.

]]> arya@projectsegfau.lt (Arya Kiran) 2023/04/17/1 https://www.evalapply.org/posts/cold-restart-total-outage/index.html https://www.evalapply.org/posts/cold-restart-total-outage/index.html Fri, 07 Apr 2023 00:00:00 GMT Aditya Athalye riff systems complexity meta https://ravidwivedi.in/posts/debian-packaging-removing-git-from-gemspec/ https://ravidwivedi.in/posts/debian-packaging-removing-git-from-gemspec/ Tue, 04 Apr 2023 20:18:53 GMT Recently, I was packaging silent_stream for Debian. sbuild was showing error because there was git in gemspec file. Specifically, this line

`git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(tests|spec|features)/}) }

in this file.

Thanks to Vinay who helped me in fixing this. It took some time and it is common to run into this error, I was told. So, I am documenting what worked for me.

We will use quilt and Rapahel has written a good article on how to use it to patch. You will need to setup quilt first which is explained in the article.

—————Run all these commands in debian unstable environment—————————–

1. Since I had already pushed my changes to salsa.debian.org, I had to clone the repository. We do this by using:

   gbp clone --pristine-tar git@salsa.debian.org:ruby-team/ruby-silent-stream.git

   cd into the repository just cloned

   cd ruby-silent-stream

2. Now, insert a new empty patch

   quilt new remove-git-in-gemspec.patch

3. Tell quilt that you intend to modify files

   quilt add remove-git-in-gemspec.patch

4. Edit the file which requires to be fixed. In our case it is silent_stream.gemspec file

   vim silent_stream.gemspec

5. Delete lines 50,51,52 in this file

6. Replace those lines with:

   spec.files = Dir.glob("**/*")

   and save the file.

7. To generate the patch, run

   quilt refresh

8. Add metadata to your patch header

   quilt header --dep3 -e

9. Add debian/patches directory to git staging area:

   git add debian/patches

10. Restore gemspec file.

    git restore silent_stream.gemspec

11. Run debclean

    debclean

12. Before running sbuild, I had to import tar files in parent directory

    uscan --verbose -dd --download-current-version

13. Check if sbuild is successful.

    sbuild -d unstable (where to run this command depends on how you setup sbuild and your debian unstable)

—————–Run above commands in debian unstable environment————

If the sbuild was successful, then commit and push your changes. I usually commit in my host system as signing the commits does not work for me inside chroot or the unstable environment I setup using systemd-nspawn.

git commit -S -m "add remove-git-in-gemspec.patch"

So, that’s it for this tutorial. Meet you in the next post.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/important-question-not-asked-prav/ https://ravidwivedi.in/posts/important-question-not-asked-prav/ Fri, 24 Mar 2023 08:05:25 GMT Earlier this month, I gave a presentation introducing Prav along with my friend Arun. Many people got curious and asked various questions about the project. However, I was surprised that nobody inquired about the operation of a privacy messaging service in India, especially considering the weakening privacy laws in the country as highlighted in this article.

The conference where I presented had at least two talks/discussions on the policy/law aspects of technology. Even outside the conference, I did not encounter this question, which I believe is crucial. It seems that people are not yet fully aware of the implications of the new IT rules.

Update on 27 May 2023: This question was later raised on social media, reminding me of a similar question asked here before I wrote this post.

]]> Ravi Dwivedi https://aryak.me/blog/01-knot.html https://aryak.me/blog/01-knot.html Wed, 01 Mar 2023 12:39:46 GMT slave sync, RFC2136 (for automatic dns-based certs in caddy and such) and GeoDNS, which makes the server give an IP closest to the user. I assume you have two Debian based systems, with port 53 (tcp+udp) forwarded. Installing Knot This guide covers debian, but instructions for other distributions can be found on the download page Run the following on both the master and the slave: apt-get -y install apt-transport-https lsb-release ca-certificates wget wget -O /usr/share/keyrings/knot.gpg https://deb.knot-dns.cz/apt.gpg sh -c 'echo "deb [signed-by=/usr/share/keyrings/knot.gpg] https://deb.knot-dns.cz/knot-latest/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/knot-latest.list' apt-get update apt-get install knot knot-dnsutils Basic Configuration knot.conf Now, to add in the basic configuration, overwrite the /etc/knot/knot.conf file on the master with the following text: server: rundir: "/run/knot" user: knot:knot listen: 0.0.0.0@53 log: - target: syslog any: info database: storage: "/var/lib/knot" remote: - id: secondary address: your.other.servers.ip@53 acl: - id: acl_secondary address: your.other.servers.ip action: transfer template: - id: default storage: "/etc/knot/zones" file: "%s.zone" semantic-checks: on # Don't override zonefile zonefile-sync: -1 zonefile-load: difference-no-serial journal-content: all zone: - domain: your.domain notify: secondary acl: acl_secondary On the slave, overwrite the same file with the following text: server: rundir: "/run/knot" user: knot:knot listen: 0.0.0.0@53 log: - target: syslog any: info database: storage: "/var/lib/knot" remote: - id: primary address: your.main.servers.ip@53 acl: - id: acl_primary address: your.main.servers.ip action: notify template: - id: default storage: "/etc/knot/zones" file: "%s.zone" zone: - domain: your.domain master: primary acl: acl_primary Zonefile At this point you need to create /etc/knot/zones on both the master and slave as that is where the zonefiles will be stored. Now create a file named your.domain.zone in the directory on the master alone and add the following text to it: $ORIGIN your.domain. ; 'default' domain as FQDN for this zone $TTL 3600 ; default time-to-live for this zone your.domain. IN SOA ns1.your.domain. ns2.your.domain. ( YYYYMMDD01 ;Serial 14400 ;Refresh 3600 ;Retry 1209600 ;Expire 3600 ;Negative response caching TTL ) @ IN NS ns1.your.domain. @ IN NS ns2.your.domain. ns1 A your.main.servers.ip ns2 A your.other.servers.ip @ A your.main.servers.ip ; PTR Records (for mailservers) your.ip.in.reverse.in-addr.arpa. PTR mail.your.domain. At this point, you can run systemctl restart knot (restart since its a major change) on both nodes. Updating the zonefile Updating the zonefile is not hard. Since we set the zonefile-load to difference-no-serial, we do not need to increment the serial as it will automatically be computed when knotc reload is run. All records must be before the PTR in the zonefile. Wildcards can be done with the hostname as * If the hostname ends with ., it must include your.domain, that is something.your.domain will be something.your.domain. If the hostname does not end with a ., it is relative to the domain, that is something.your.domain will just be something. Always use tabs, not spaces. Token-based zonefile authentication At the moment, the authentication for sending and receiving the zonefile is completely based on the IP address, which is not very secure. To remediate this, we can use token-based authentication. First, you need to generate a key with keymgr -t zonesync hmac-sha256 This is the key that will authenticate the zone transfers. You can copy-paste the output it gives you into your knot.conf, above the remote section on both master and slave. Now, you need to add key: zonesync to the remote, acl_primary/secondary sections on both master and slave. At this point, you can run systemctl restart knot, and all the syncs will be more secure! DNSSEC DNSSEC is an extension to DNS designed to protect applications using DNS from accepting forged or manipulated DNS data by using zone signing. Enabling DNSSEC on knot is as simple as adding the following line to the template section: template: ... dnssec-signing: on ... At this point, you need to upload the DS record to your Registry. This is usually done through your registrar’s WebUI. The DNSSEC-enabled DNS servers use the DS record from your domain registry to validate your records. You can get your DS record with the following command: keymgr your.domain ds The output will look something like this: 54674 13 2 E28E3DB78E5517A577353A43799AD14EC044720BAE4906D134F5EA40 74AC0287 In the example given: Key tag - 54674 Algorithm - 13 Digest Type - 2 Digest - E28E3…287 (omit space) On namecheap, you add this at Advanced DNS -> DNSSEC You can check if your DNSSEC is working properly at DNSViz and DNSSEC-Analyzer. RFC2136 RFC2136 is an RFC that allows dynamic updates for DNS. This works completely over DNS and does not require a special API. To set it up, you need to create an ACL as follows: acl: ... - id: acl_dynupdates address: [an.authorized.ip.addr, another.authorized.ip.addr] action: update ... zone: - domain: your.domain notify: secondary acl: [acl_secondary, acl_dynupdates] You can also add token-based auth by generating another key with keymgr -t rfc2136 hmac-sha256 You can add it to the config as follows: key: ... - id: rfc2136 algorithm: hmac-sha256 secret: xxx ... acl: ... - id: acl_dynupdates address: [an.authorized.ip.addr, another.authorized.ip.addr] action: update key: rfc2136 After this, you can run systemctl restart knot to apply the changes. Caddy Caddy is a modern webserver which supports automatic cert generation from letsencrypt/zerossl with acme. It uses http-01 for the challenge by default, but can use a dns challenge too. For the DNS challenge, you first need to install the RFC2136 DNS plugin with the following command: xcaddy build --with github.com/caddy-dns/rfc2136@master Now, add the following lines to the top of your caddyfile { acme_dns rfc2136 { key_name "rfc2136" key_alg "hmac-sha256" key "xxx" server "your.main.servers.ip:53" } acme_ca https://acme-v02.api.letsencrypt.org/directory } Now, all certs can be generated using the DNS challenge! This is especially useful in a GeoDNS environment. GeoDNS GeoDNS allows geographical split horizon based on a GeoIP database, such as Maxmind’s free GeoLite2. Firstly, you need to procure the GeoLite2 database from Maxmind. Due to policy changes, you now need to signup on Maxmind’s website in order to get access. However, older GeoIP DBs can still be found in many places, including distribution package repositories. Once you procure your copy of GeoLite2, you need to install the GeoIP module for knot-dns on both master and slave. On Debian, the package’s name is knot-module-geoip After installing the module, you can add it to the knot.conf (must be before zone section however): mod-geoip: - id: geo config-file: "/etc/knot/geo.conf" mode: geodb geodb-file: "/var/lib/knot/GeoLite2-City.mmdb" geodb-key: [ continent/code, country/iso_code, city/names/en ] You also need to include the GeoIP module for your domain. You can do that by adding this line to your domain’s zone section: zone: - domain: your.domain ... module: mod-geoip/geo Now, you need to configure the GeoDNS. To do so, create a file called /etc/knot/geo.conf with the following: geodnsubdom.your.domain: - geo: "*;*;*" # Fallback incase DNS server doesn't send ECS A: your.main.servers.ip TXT: "Worldwide" - geo: "EU;*;*" # Europe A: your.europe.servers.ip TXT: "Europe" ... However, the file needs to be manually synced to the slave on every update. It is also painful to use if you have multiple root subdomains you want to use GeoDNS with. Due to these reasons, I made a kinda hacky script to remediate this: #!/usr/bin/env bash geoconf=/etc/knot/geo.conf remote='geodns@other.servers.ip' printf '' > $geoconf for i in $(> $geoconf sed -i "s/REPLACEME/${i}/" $geoconf done scp $geoconf "${remote}":/var/geo.conf ssh $remote "sudo systemctl restart knot" systemctl restart knot In order to allow the unprivileged user on the slave to restart knot, I used a sudo ALLOW_CMDS flag: Cmnd_Alias KNOT_CMDS = /usr/bin/systemctl restart knot geodns ALL=(ALL) NOPASSWD: KNOT_CMDS And thats it. Thanks for reading!]]> Knot DNS is one of the easier to setup authoritative dns servers out there, made by NIC.CZ.

In this tutorial I’ll show how to setup Knot with DNSSEC, authenticated master -> slave sync, RFC2136 (for automatic dns-based certs in caddy and such) and GeoDNS, which makes the server give an IP closest to the user.

I assume you have two Debian based systems, with port 53 (tcp+udp) forwarded.

Installing Knot

This guide covers debian, but instructions for other distributions can be found on the download page Run the following on both the master and the slave:

apt-get -y install apt-transport-https lsb-release ca-certificates wget
wget -O /usr/share/keyrings/knot.gpg https://deb.knot-dns.cz/apt.gpg
sh -c 'echo "deb [signed-by=/usr/share/keyrings/knot.gpg] https://deb.knot-dns.cz/knot-latest/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/knot-latest.list'
apt-get update
apt-get install knot knot-dnsutils

Basic Configuration

knot.conf

Now, to add in the basic configuration, overwrite the /etc/knot/knot.conf file on the master with the following text:

server:
    rundir: "/run/knot"
    user: knot:knot
    listen: 0.0.0.0@53
log:
  - target: syslog
    any: info
database:
    storage: "/var/lib/knot"
remote:
  - id: secondary
    address: your.other.servers.ip@53
acl:
  - id: acl_secondary
    address: your.other.servers.ip
    action: transfer
template:
  - id: default
    storage: "/etc/knot/zones"
    file: "%s.zone"
    semantic-checks: on
    # Don't override zonefile
    zonefile-sync: -1
    zonefile-load: difference-no-serial
    journal-content: all
zone:
  - domain: your.domain
    notify: secondary
    acl: acl_secondary

On the slave, overwrite the same file with the following text:

server:
    rundir: "/run/knot"
    user: knot:knot
    listen: 0.0.0.0@53
log:
  - target: syslog
    any: info
database:
    storage: "/var/lib/knot"
remote:
  - id: primary
    address: your.main.servers.ip@53
acl:
  - id: acl_primary
    address: your.main.servers.ip
    action: notify
template:
  - id: default
    storage: "/etc/knot/zones"
    file: "%s.zone"
zone:
  - domain: your.domain
    master: primary
    acl: acl_primary

Zonefile

At this point you need to create /etc/knot/zones on both the master and slave as that is where the zonefiles will be stored. Now create a file named your.domain.zone in the directory on the master alone and add the following text to it:

$ORIGIN your.domain. ; 'default' domain as FQDN for this zone
$TTL 3600 ; default time-to-live for this zone

your.domain.   IN  SOA     ns1.your.domain. ns2.your.domain. (
        YYYYMMDD01  ;Serial
        14400       ;Refresh
        3600        ;Retry
        1209600     ;Expire
        3600        ;Negative response caching TTL
)
@   IN  NS  ns1.your.domain.
@   IN  NS  ns2.your.domain.
ns1 A   your.main.servers.ip
ns2 A   your.other.servers.ip
@   A   your.main.servers.ip

; PTR Records (for mailservers)
your.ip.in.reverse.in-addr.arpa.    PTR mail.your.domain.

At this point, you can run systemctl restart knot (restart since its a major change) on both nodes.

Updating the zonefile

Updating the zonefile is not hard.

• Since we set the zonefile-load to difference-no-serial, we do not need to increment the serial as it will automatically be computed when knotc reload is run.
• All records must be before the PTR in the zonefile.
• Wildcards can be done with the hostname as *
• If the hostname ends with ., it must include your.domain, that is something.your.domain will be something.your.domain.
• If the hostname does not end with a ., it is relative to the domain, that is something.your.domain will just be something.
• Always use tabs, not spaces.

Token-based zonefile authentication

At the moment, the authentication for sending and receiving the zonefile is completely based on the IP address, which is not very secure.

To remediate this, we can use token-based authentication.

First, you need to generate a key with keymgr -t zonesync hmac-sha256

This is the key that will authenticate the zone transfers.

You can copy-paste the output it gives you into your knot.conf, above the remote section on both master and slave.

Now, you need to add key: zonesync to the remote, acl_primary/secondary sections on both master and slave.

At this point, you can run systemctl restart knot, and all the syncs will be more secure!

DNSSEC

DNSSEC is an extension to DNS designed to protect applications using DNS from accepting forged or manipulated DNS data by using zone signing.

Enabling DNSSEC on knot is as simple as adding the following line to the template section:

template:
    ...
    dnssec-signing: on
    ...

At this point, you need to upload the DS record to your Registry. This is usually done through your registrar’s WebUI.

The DNSSEC-enabled DNS servers use the DS record from your domain registry to validate your records.

You can get your DS record with the following command:

keymgr your.domain ds

The output will look something like this:

54674 13 2 E28E3DB78E5517A577353A43799AD14EC044720BAE4906D134F5EA40 74AC0287

In the example given:

• Key tag - 54674
• Algorithm - 13
• Digest Type - 2
• Digest - E28E3…287 (omit space)

On namecheap, you add this at Advanced DNS -> DNSSEC

You can check if your DNSSEC is working properly at DNSViz and DNSSEC-Analyzer.

RFC2136

RFC2136 is an RFC that allows dynamic updates for DNS.

This works completely over DNS and does not require a special API.

To set it up, you need to create an ACL as follows:

acl:
    ...
    - id: acl_dynupdates
    address: [an.authorized.ip.addr, another.authorized.ip.addr]
    action: update
...
zone:
  - domain: your.domain
    notify: secondary
    acl: [acl_secondary, acl_dynupdates]

You can also add token-based auth by generating another key with keymgr -t rfc2136 hmac-sha256

You can add it to the config as follows:

key:
    ...
    - id: rfc2136
    algorithm: hmac-sha256
    secret: xxx
...
acl:
    ...
    - id: acl_dynupdates
    address: [an.authorized.ip.addr, another.authorized.ip.addr]
    action: update
    key: rfc2136

After this, you can run systemctl restart knot to apply the changes.

Caddy

Caddy is a modern webserver which supports automatic cert generation from letsencrypt/zerossl with acme.

It uses http-01 for the challenge by default, but can use a dns challenge too.

For the DNS challenge, you first need to install the RFC2136 DNS plugin with the following command:

xcaddy build --with github.com/caddy-dns/rfc2136@master

Now, add the following lines to the top of your caddyfile

{
    acme_dns rfc2136 {
        key_name "rfc2136"
        key_alg "hmac-sha256"
        key "xxx"
        server "your.main.servers.ip:53"
    }
    acme_ca https://acme-v02.api.letsencrypt.org/directory
}

Now, all certs can be generated using the DNS challenge!

This is especially useful in a GeoDNS environment.

GeoDNS

GeoDNS allows geographical split horizon based on a GeoIP database, such as Maxmind’s free GeoLite2.

Firstly, you need to procure the GeoLite2 database from Maxmind.

Due to policy changes, you now need to signup on Maxmind’s website in order to get access.

However, older GeoIP DBs can still be found in many places, including distribution package repositories.

Once you procure your copy of GeoLite2, you need to install the GeoIP module for knot-dns on both master and slave.

On Debian, the package’s name is knot-module-geoip

After installing the module, you can add it to the knot.conf (must be before zone section however):

mod-geoip:
  - id: geo 
    config-file: "/etc/knot/geo.conf"
    mode: geodb
    geodb-file: "/var/lib/knot/GeoLite2-City.mmdb"
    geodb-key: [ continent/code, country/iso_code, city/names/en ]

You also need to include the GeoIP module for your domain. You can do that by adding this line to your domain’s zone section:

zone:
    - domain: your.domain
      ...
      module: mod-geoip/geo

Now, you need to configure the GeoDNS.

To do so, create a file called /etc/knot/geo.conf with the following:

geodnsubdom.your.domain:
  - geo: "*;*;*" # Fallback incase DNS server doesn't send ECS
    A: your.main.servers.ip
    TXT: "Worldwide"
  - geo: "EU;*;*" # Europe
    A: your.europe.servers.ip
    TXT: "Europe"
    ...

However, the file needs to be manually synced to the slave on every update.

It is also painful to use if you have multiple root subdomains you want to use GeoDNS with.

Due to these reasons, I made a kinda hacky script to remediate this:

#!/usr/bin/env bash
geoconf=/etc/knot/geo.conf
remote='geodns@other.servers.ip'
printf '' > $geoconf
for i in $(</etc/knot/geodnsdomains); do
    cat /etc/knot/geodnstemplate >> $geoconf
    sed -i "s/REPLACEME/${i}/" $geoconf
done
scp $geoconf "${remote}":/var/geo.conf
ssh $remote "sudo systemctl restart knot"
systemctl restart knot

In order to allow the unprivileged user on the slave to restart knot, I used a sudo ALLOW_CMDS flag:

Cmnd_Alias KNOT_CMDS = /usr/bin/systemctl restart knot
geodns ALL=(ALL) NOPASSWD: KNOT_CMDS

And thats it. Thanks for reading!

]]> arya@projectsegfau.lt (Arya Kiran) 2023/03/01/3 https://www.divyamohan.com/blog-open-source-sustainability/ https://www.divyamohan.com/blog-open-source-sustainability/ Sat, 18 Feb 2023 13:55:22 GMT Remember this xkcd comic?

Over the past couple of years, open source sustainability has been a recurrent theme in discussions with friends and colleagues alike.

Whether it be watching the core-js incident play out during the last week or the left-pad incident in 2016, it is evident that there is a huge sustainability problem in the open source JavaScript community.

But you'd be grossly mistaken if you thought this problem was specific to the JavaScript community alone. The Christmas gift of the log4shell incident in 2021 was my first brush with being personally responsible for rallying troops to mitigate the vulnerability at my workplace. I was employed with a bank at the time and clearly remember the panic and scurrying that ensued after the vulnerability was announced. I also remember how almost all consumers had their bytes and thought pieces ready for promises of support. Has much changed? One look at the project team tells you that we're still staring at a massive imbalance in the consumer/publisher ratio when it comes to open source.

I still don't get it. So what exactly is the problem?

Despite using open source software directly or indirectly in every aspect of our lives, many folks outside the OSS ecosystem have no idea about the enormous efforts behind the scenes to keep the machine well-oiled.

Why do I say that so confidently? Because I was one of those people, too, till 2019. I worked on Linux systems and was responsible for middleware administration for several apps built atop open source software such as Apache, JBoss, etc. Yet, my understanding of open source was limited to it being freely available. I didn't delve further because what was the point? It's not like my employers cared about it much, either.

Not until, of course, some bug affected the software, and all hell would break loose. Efforts would then be rallied to mitigate the bug at our end, and we'd wait until a patch was published to roll it out. All so simple, right? Wrong.

At that point, what was invisible to me were the toil and person hours behind releasing those patches, often at breakneck speed and under extreme pressure.

That's bad. But how does it matter?

The invisible toil & person-hours? They're expended by humans like you and me sitting behind a computer with families to feed and rent to pay.

How many of those folks receive adequate compensation for their contributions?

For context, everyone in the IT industry has used or at least heard of OpenSSL.

If you haven't, here's me quoting Wikipedia directly,

OpenSSL is a software library for applications that provide secure communications over computer networks against eavesdropping or the need to identify the party at the other end. It is widely used by Internet servers, including the majority of HTTPS websites.

From that definition, it looks like a pretty important piece of software, right? Guess how many core developers were maintaining the project during the Heartbleed vulnerability that affected almost everyone using the internet? Four.

One of the core developers, Steven Henson, was the only person working full-time on the project. Additionally, according to this article, the group received just $2000 for the upkeep of the project before the Heartbleed incident.

Overworked maintainers & understaffed projects are the foundations that open source and, thereby, most of your daily drivers are built on.

Why is it MY problem?

I often envision the open source model as a seesaw. That analogy is me dumbing it down because I am still navigating this ecosystem as we speak, and I have a lot to learn!

But if you stick with the analogy, you'll notice that it does play out well from a visualization perspective.

Assume we have the consumers on one side & the maintainers on the other side of the seesaw. It is idealistic to assume that we started at an equilibrium.

However, even if we did, we are currently in a situation where project maintainers are now staring at a massive imbalance with

• More customized demands due to every consumer working off their respective fork.
• Lack of contributors to help with the incoming volume of requests
• Lack of commensurate support and compensation from companies that employ the maintainers and contributors
• Eventual burnout

When that snowballs, it is only human to want to step away. After all, despite loving what they do, there's only so much a person can give before their vessel is empty.

Where does that leave us as an industry? Lesser collaboration, a slowdown in innovation, and a lot of paywalled software. None of the things that we take for granted today will continue to exist.

This may sound a bit dramatic, but it's every bit true as an eventuality if the current situation continues.

The way ahead

In 2021, I wrote a piece for Container Solutions about how corporate open source consumers could be better citizens days before the log4shell vulnerability was discovered. It's been more than a year, yet most of the measures I suggested then are still relevant today.

When I wrote that article, Open Source Program Offices (or OSPOs) had just begun appearing on my radar. I want to attribute that to my lack of knowledge and awareness of the goings-on within the ecosystem.

But I'd also be the first to admit today that there's been quiet a bit of traction in adopting OSPOs across the industry since then. So that's probably one of the things I'd recommend learning more about in the context of your organization if I were to write the article today.

But to sum it all up, Floor Drees, Staff Community Program Manager at Aiven, states it more beautifully than I ever could in this TNS piece,

"Discussions around the sustainability of open source are hard, but they’re necessary. FOSS is ubiquitous, it’s omnipresent, yet we’re still struggling to live with open source in a healthy, safe, and productive way.

As Floor noted, we need to understand the risks and then help open source, well, help open source.

]]> Divya Mohan Open Source Blog https://mrkaran.dev/posts/missing-duplicate-logs/ https://mrkaran.dev/posts/missing-duplicate-logs/ Thu, 16 Feb 2023 08:27:13 GMT { return { level: label }; }, }, } pilog = pino(opts, pino.destination(`${__dirname}/app.log`)) pilog.info('this is a first message') setTimeout(() => pilog.info('this message should get logged'), 10000) pilog.info('this message will be recorded as well') This snippet logs to app.log. {"level":"info","time":"2023-02-16T06:45:04.031Z","pid":206573,"hostname":"pop-os","name":"outbound_logger","msg":"this is a first message"} {"level":"info","time":"2023-02-16T06:45:04.031Z","pid":206573,"hostname":"pop-os","name":"outbound_logger","msg":"this message will be recorded as well"} During the 10s time interval, I deleted the file from the disk rm app.log to mimic the behaviour of remove_after_secs. I expected the file to get re-created and this message should get logged logged by the above script. However, that didn’t happen. The script didn’t complain about a missing file, either. I was perplexed and did some google-fu and found the following via Stackoverflow: The writes actually do not fail.When you delete a file that is open in another program you are deleting a named link to that file’s inode. The program that has it open still points to that inode. It will happily keep writing to it, actually writing to disk. Only now you don’t have a way to look it at, because you deleted the named reference to it. (If there were other references, e.g. hard links, you would still be able to!). This is exactly what was happening in production. When vector deleted the file (as configured via remove_after_secs ), the plugin didn’t know about it and kept writing to the same inode. This was a major TIL moment for me. Fix: The fix was simple enough; I removed remove_after_secs from Vector’s config. To address the problem of the file not growing unbounded forever, I created a logrotate config: /opt/app/logs/app.log { daily rotate 15 dateext dateformat -%Y-%m-%d.log delaycompress compress notifempty missingok copytruncate } Some notes: copytruncate is useful in this context. It copies the existing file to a new one, which now becomes a stale one. The current (active) file will be truncated to zero bytes. E.g., if app.log it is rotated, logrotate will copy the file to a new file app-2023-02-05.log and then truncate the existing one to zero bytes. delaycompress will not compress the logs until the next rotation happens. This is useful if vector hasn’t finished processing the log and can continue to do that. Duplicate Logs# Now after fixing the case of missing logs, I found myself in the opposite problem - now the logs were duplicated on Clickhouse. My mental state at that moment couldn’t be more accurately described than this meme: To add more context, before developing my plugin for logging email delivery in Haraka, we used another plugin (acharkizakaria/haraka-plugin-accounting-files) to get these logs. This plugin records the metadata to CSV files. Still, there were some issues in properly escaping the subject lines (if the subject had a comma, that was incorrectly parsed); hence, the log file had inconsistent output. To address these issues, I found writing another plugin from scratch that outputs to a fixed JSON schema is better. As seen above, vector’s file source was configured like the following for reading CSV files. The only change here is that remove_after_secs is gone after fixing issue #1. [sources.outbound_logger] type = "file" include = ["/var/log/haraka/outbound/*.log"] read_from = "beginning" fingerprint.strategy = "device_and_inode" Vector “fingerprints” the file source, as it keeps the checkpoint of how many bytes it has read for each file in its own “disk buffer”. This buffer is helpful if Vector crashes so it can restart reading the file from where it last stopped. There are two strategies for fingerprinting that vector uses: The checksum strategy uses CRC check on the first N lines of the file. The device_and_inode strategy uses the disk’s actual inode location to identify the file uniquely. As I was using a different plugin which logged to a CSV file, the checksum strategy did not work in my context. Since vector fingerprints, the first few bytes (usually just enough for a header of a CSV), all the CSV files in that disk would have the same title, and Vector would not read all of them. To work around this, I changed the fingerprint.starategy = "device_and_inode" so Vector uniquely identifies all CSV files by their inode path. (In hindsight, I should have just used checksum with a higher value of fingerprint.lines value.) The mistake this time was when I switched to a JSON log file, I continued with the device_and_inode strategy. This isn’t a problem if there is no log rotation setup. Since I did configure logrotate to fix issue #1, as you would have guessed, copytruncate created another log file, and because I was using device_and_inode strategy, vector thought this was a “new” file to be watched and processed. So now I had duplicate entries from this new file, which is technically just an older log-rotated file. The fix: [sources.outbound_logger.fingerprint] lines = 14 strategy = "checksum" ignored_header_bytes = 2048 I switched back to the default checksum strategy and adjusted the thresholds for lines/header bytes to account for JSON logs. The same is also documented very clearly in Vector and it was my RTFM moment. This strategy avoids the common pitfalls associated with using device and inode names since inode names can be reused across files. This enables Vector to properly tail files across various rotation strategies. Phew! I am glad after these fixes; vector is durably and reliably processing all the logs and logrotate is happily working in conjunction as well. I hope documenting my learnings about these production issues would help someone with the same problems. Fin!]]> At work, we use a Vector pipeline for processing and shipping logs to Clickhouse. We also self-host our SMTP servers and recently started using Haraka SMTP. While Haraka is excellent in raw performance and throughput, it needed an external logging plugin for audit and compliance purposes. I wrote haraka-plugin-outbound-logger to log basic metadata like timestamps/subject/SMTP response in a JSON file.

The plan was to dump these logs into a file and use Vector’s file source for reading them and doing the further transformation. However, things went differently than I had planned. There were mainly two issues propped up due to bad Vector configuration.

Missing logs#

The vector configuration to read the file looked like this:

[sources.outbound_logger]
type = "file"
include = ["/var/log/haraka/outbound/*.log"]
read_from = "beginning"
# Remove the files after 24 hours. Vector must have permission to delete these files.
remove_after_secs = 86400
fingerprint.strategy = "device_and_inode"

Vector has a handy configuration of automagically deleting the file if the file hasn’t received any new write in the configured time interval. So remove_after_secs=86400 specifies if the file hasn’t had any new writes since 24h, it can delete it. It made sense to configure because our workload was a shorter process. It was a batch job done once every N days.

When the file didn’t receive any new writes after 24h, vector deleted the file as expected. However, the plugin continued logging into the same file handler, even for newer batch jobs. As a result, the file didn’t receive any new logs and was empty.

I created a minimal POC to reproduce this seemingly strange issue:

var pino = require('pino');

// Initialise pino js logger and inject in the plugin context.
var opts = {
    name: 'outbound_logger',
    level: 'debug',
    // uses the ISO time format.
    timestamp: pino.stdTimeFunctions.isoTime,
    formatters: {
        level: (label) => {
            return { level: label };
        },
    },
}

pilog = pino(opts, pino.destination(`${__dirname}/app.log`))

pilog.info('this is a first message')
setTimeout(() => pilog.info('this message should get logged'), 10000)
pilog.info('this message will be recorded as well')

This snippet logs to app.log.

{"level":"info","time":"2023-02-16T06:45:04.031Z","pid":206573,"hostname":"pop-os","name":"outbound_logger","msg":"this is a first message"}
{"level":"info","time":"2023-02-16T06:45:04.031Z","pid":206573,"hostname":"pop-os","name":"outbound_logger","msg":"this message will be recorded as well"}

During the 10s time interval, I deleted the file from the disk rm app.log to mimic the behaviour of remove_after_secs. I expected the file to get re-created and this message should get logged logged by the above script.

However, that didn’t happen. The script didn’t complain about a missing file, either. I was perplexed and did some google-fu and found the following via Stackoverflow:

The writes actually do not fail.When you delete a file that is open in another program you are deleting a named link to that file’s inode. The program that has it open still points to that inode. It will happily keep writing to it, actually writing to disk. Only now you don’t have a way to look it at, because you deleted the named reference to it. (If there were other references, e.g. hard links, you would still be able to!).

This is exactly what was happening in production. When vector deleted the file (as configured via remove_after_secs ), the plugin didn’t know about it and kept writing to the same inode. This was a major TIL moment for me.

Fix: The fix was simple enough; I removed remove_after_secs from Vector’s config. To address the problem of the file not growing unbounded forever, I created a logrotate config:

/opt/app/logs/app.log {
    daily
    rotate 15
    dateext
    dateformat -%Y-%m-%d.log
    delaycompress
    compress
    notifempty
    missingok
    copytruncate
}

Some notes:

• copytruncate is useful in this context. It copies the existing file to a new one, which now becomes a stale one. The current (active) file will be truncated to zero bytes. E.g., if app.log it is rotated, logrotate will copy the file to a new file app-2023-02-05.log and then truncate the existing one to zero bytes.
• delaycompress will not compress the logs until the next rotation happens. This is useful if vector hasn’t finished processing the log and can continue to do that.

Duplicate Logs#

Now after fixing the case of missing logs, I found myself in the opposite problem - now the logs were duplicated on Clickhouse.

My mental state at that moment couldn’t be more accurately described than this meme:

To add more context, before developing my plugin for logging email delivery in Haraka, we used another plugin (acharkizakaria/haraka-plugin-accounting-files) to get these logs. This plugin records the metadata to CSV files. Still, there were some issues in properly escaping the subject lines (if the subject had a comma, that was incorrectly parsed); hence, the log file had inconsistent output. To address these issues, I found writing another plugin from scratch that outputs to a fixed JSON schema is better.

As seen above, vector’s file source was configured like the following for reading CSV files. The only change here is that remove_after_secs is gone after fixing issue #1.

[sources.outbound_logger]
type = "file"
include = ["/var/log/haraka/outbound/*.log"]
read_from = "beginning"
fingerprint.strategy = "device_and_inode"

Vector “fingerprints” the file source, as it keeps the checkpoint of how many bytes it has read for each file in its own “disk buffer”. This buffer is helpful if Vector crashes so it can restart reading the file from where it last stopped.

There are two strategies for fingerprinting that vector uses:

• The checksum strategy uses CRC check on the first N lines of the file.
• The device_and_inode strategy uses the disk’s actual inode location to identify the file uniquely.

As I was using a different plugin which logged to a CSV file, the checksum strategy did not work in my context. Since vector fingerprints, the first few bytes (usually just enough for a header of a CSV), all the CSV files in that disk would have the same title, and Vector would not read all of them. To work around this, I changed the fingerprint.starategy = "device_and_inode" so Vector uniquely identifies all CSV files by their inode path. (In hindsight, I should have just used checksum with a higher value of fingerprint.lines value.)

The mistake this time was when I switched to a JSON log file, I continued with the device_and_inode strategy. This isn’t a problem if there is no log rotation setup. Since I did configure logrotate to fix issue #1, as you would have guessed, copytruncate created another log file, and because I was using device_and_inode strategy, vector thought this was a “new” file to be watched and processed. So now I had duplicate entries from this new file, which is technically just an older log-rotated file.

The fix:

[sources.outbound_logger.fingerprint]
lines = 14
strategy = "checksum"
ignored_header_bytes = 2048

I switched back to the default checksum strategy and adjusted the thresholds for lines/header bytes to account for JSON logs. The same is also documented very clearly in Vector and it was my RTFM moment.

This strategy avoids the common pitfalls associated with using device and inode names since inode names can be reused across files. This enables Vector to properly tail files across various rotation strategies.

Phew! I am glad after these fixes; vector is durably and reliably processing all the logs and logrotate is happily working in conjunction as well. I hope documenting my learnings about these production issues would help someone with the same problems.

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/structured-logging-in-go-with-slog/ https://mrkaran.dev/posts/structured-logging-in-go-with-slog/ Wed, 15 Feb 2023 07:29:53 GMT A few months ago, a proposal for adding a structured logging library in Go was introduced by Jonathan Amsterdam. At present, Go has a minimal and bare-bones log package which works all right for basic use cases. However, the current library has a few shortcomings that this proposal aims to solve:

• Emitting logs with different severity/levels
• Structured output: Makes parsing of logs harder
• Logging a set of common fields/attributes
• Difficult to have a log object inside libraries as each service could have its log implementation.

As a result, many code bases have their wrappers around the log package. Additionally, there are plenty of 3rd party libraries to choose from - including logf (which my work colleagues and I built at Zerodha).

This article is about how to get started with slog for logging in Go applications.

NOTE

Since slog is currently in the proposal state and hasn’t yet merged in the official library, the API could change in future.

Architecture of slog#

At a higher level, slog contains three main entities:

• Logger: The user-facing API for interacting with slog. All the public methods are defined on the Logger object.
• Record: Contains information about the log event itself. A standard record will have timestamp, level and message fields as default. Additional attributes and metadata like caller info can be added to the Record.
• Handlers: A handler is an interface implementation. The Logger object passes the Record to a handler, and the handler can choose whatever it wants to do with the Record. This is a common approach in Go libraries, where a “provider” can be abstracted in handling that task. Currently, slog ships with two handlers: JSON and logfmt. Some projects have also created handlers for zap/logrus (popular 3rd party libraries).

Initialization#

This snippet initializes a Text Handler, which produces logfmt format messages on os.Stdout.

package main

import (
	"os"

	"golang.org/x/exp/slog"
)

func main() {
	log := slog.New(slog.NewTextHandler(os.Stdout))
	log.Info("Hello world")

	fakeErr := os.ErrNotExist
	log.Error("something went wrong", fakeErr, "file", "/tmp/abc.txt")
}

Log output:

time=2023-02-15T19:58:10.615+05:30 level=INFO msg="Hello world"
time=2023-02-15T19:58:10.615+05:30 level=ERROR msg="something went wrong" file=/tmp/abc.txt err="file does not exist"

Customizing#

You’ll notice that the caller information isn’t exposed by default. The reason could be that finding the stack trace of the calling line is a bit expensive operation. However, for libraries/apps which need it can do that by customizing the handler:

func main() {
	handler := slog.HandlerOptions{AddSource: true}
	log := slog.New(handler.NewTextHandler(os.Stdout))

	log.Info("Hello world")
}

Log Output:

time=2023-02-15T12:17:53.742+05:30 level=INFO source=/home/karan/Code/Personal/slog-examples/main.go:14 msg="Hello world"

Attributes#

Sometimes, it’s helpful to append specific metadata to each log line which will help in aggregating/filtering with a central log-collecting agent. E.g., you can export a component key for each sub-service of your primary application.

func main() {
	log := slog.New(slog.NewTextHandler(os.Stdout)).With("component", "demo")
	log.Info("Hello world")
}

Log Output:

time=2023-02-15T12:21:50.231+05:30 level=INFO msg="Hello world" component=demo

Nested Keys#

So far, we’ve seen flat keys in the log message. It may be helpful to group together specific keys together and form a nested object. In JSON, that would mean a top-level object with different fields inside. However, in logfmt, it would-be parent.child format.To use nested keys, slog.Group can be used. This example uses http as the top-level key, and all its associated fields will be nested inside.

	log.Info("Hello world", slog.Group("http",
		slog.String("method", "GET"),
		slog.Int("status", 200),
		slog.Duration("duration", 250),
		slog.String("method", "GET"),
		slog.String("path", "/api/health")))

time=2023-02-15T12:30:43.130+05:30 level=INFO msg="Hello world" component=demo http.method=GET http.status=200 http.duration=250ns http.method=GET http.path=/api/health

Configurable Handlers#

JSON logs are daunting and tedious to read when locally developing applications. However, it’s a great fit for machine parsing of the logs. logfmt hits the sweet spot for being machine parseable and human-readable.However, thanks to the powerful “interface” implementation approach, it’s easy to switch to any handler via user-configurable methods (like config files/env variables):

package main

import (
	"os"

	"golang.org/x/exp/slog"
)

func main() {
	var (
		env     = os.Getenv("APP_ENV")
		handler slog.Handler
	)

	switch env {
	case "production":
		handler = slog.NewJSONHandler(os.Stdout)
	default:
		handler = slog.NewTextHandler(os.Stdout)
	}

	log := slog.New(handler)
	log.Info("Hello world")
}

$ go run main.go
time=2023-02-15T12:39:45.543+05:30 level=INFO msg="Hello world"
$ APP_ENV=production go run main.go
{"time":"2023-02-15T12:39:53.523477544+05:30","level":"INFO","msg":"Hello world"}

Summary#

slog is an excellent proposal, and it’s high time Go gets its official structured logging library. The API is designed to be easy to use, and a clear path is given for users wanting high-performance/zero-allocs by creating their handlers and making these performance improvements.

Fin

]]> Karan Sharma https://www.evalapply.org/posts/bad-matrix/index.html https://www.evalapply.org/posts/bad-matrix/index.html Tue, 14 Feb 2023 00:00:00 GMT Aditya Athalye riff bash https://aryak.me/blog/00-welcome.html https://aryak.me/blog/00-welcome.html Sat, 28 Jan 2023 12:39:46 GMT Hello everybody.

Welcome to my new blog!

I created this along with my website redesign and conversion to pandoc from static html.

I will try to write based content atleast once a month but no promises :P

]]> arya@projectsegfau.lt (Arya Kiran) 2023/01/28/6 https://www.evalapply.org/posts/tools-for-thought/index.html https://www.evalapply.org/posts/tools-for-thought/index.html Thu, 19 Jan 2023 00:00:00 GMT Aditya Athalye meta riff ai intelligence_augmentation tools_for_thought https://ravidwivedi.in/posts/backstory-behind-prav-app-project/ https://ravidwivedi.in/posts/backstory-behind-prav-app-project/ Wed, 18 Jan 2023 15:32:40 GMT In 2020, I uninstalled WhatsApp due to its proprietary nature and being backed by a surveillance company. In my quest to find alternatives, I came across many messengers, before finally settling on Matrix and XMPP, both of which are decentralized chat protocols built on freedom-respecting software.

But chat applications are only as good as the number of people using them, and it was not easy to onboard people to XMPP or Matrix.

Creating an account on either of them requires installing an app, followed by choosing a server, a username, and a password. This onboarding process is relatively difficult compared to more commonly-known messengers like WhatsApp and Telegram. It was suitable for privacy-conscious users, but not for the majority of people who are not so privacy-conscious, or who just found it too much effort to invest into a new app.

At some point, Praveen (of FSCI fame) introduced me to Quicksy - an XMPP client which provides two features found in WhatsApp -

1. easier registration, allowing users to register by entering a phone number and an OTP;
2. easier contact discovery, which automatically adds people to your Quicksy contact list if you have their phone number in your phone book.

These made onboarding easier, and I was successfully able to use Quicksy to onboard users.

At the time, we were also running our own XMPP services at poddery.com and disap.in. These services were run by volunteers, and there had been instances where it took us months to fix something.

Further, we found that the XMPP clients weren’t consistent in their features across platforms. For example, Kaidan didn’t have OMEMO encryption, while Gajim doesn’t support calls. So, if a user wants to use a desktop client to make XMPP calls, they need to figure out that they have to install Dino (which is only available on GNU/Linux). Having a consistent brand across all platforms helps newbies, as it might be hard for them to wrap their head around using Dino to talk to Conversations.

One day, when Praveen’s friend installed Quicksy to share pictures with him, he gave him an idea to make a business out of it. Praveen also thought it would be better to market our own XMPP service rather than Quicksy, giving us more control over the privacy policy and service operations. This is when he asked me if I would like to get involved, to which I immediately agreed.

Praveen had many years of experience running XMPP services, albeit in a volunteer setting. He thought of experimenting with paid sysadmins rather than volunteers. This is how the Prav project was born. The phone number and contact discovery part was inspired by the Quicksy app, and the idea of consistent branding across platforms was inspired by Snikket. We also added the idea of making it subscription based, which can help us fund sysadmins and allocate funds towards marketing.

A few months later, we decided to make it a cooperative so that the decision-making is democratic (one person, one vote, as opposed to companies in which each person has voting power proportional to the number of shares they hold). Nagarjuna proposed we run a beta service to attract people to pledge the cooperative membership. This gave birth to our beta app and service.

Free software gives users the freedom to adapt the software - but in practice, only programmers or resourceful entities like governments or corporations get to exercise this freedom. With the cooperative model of the Prav project, we want to empower members to collectively decide upon which features to implement, followed by raising funds for developers to implement them.

I am excited to see how this experiment will turn out.

]]> Ravi Dwivedi https://www.evalapply.org/posts/animate-text-art-javascript/index.html https://www.evalapply.org/posts/animate-text-art-javascript/index.html Mon, 16 Jan 2023 00:00:00 GMT Aditya Athalye hanukkah_of_data text_art design websites frontend javascript functional_programming web_development howto https://mrkaran.dev/posts/coredns-nomad/ https://mrkaran.dev/posts/coredns-nomad/ Wed, 04 Jan 2023 18:30:00 GMT ../../coredns-nomad Next, you can run make in coredns repository. It’ll build the binary and place it in coredns directory. You can run this binary to test your plugin. To check if the plugin indeed exists in the binary, you can use the following command ./coredns -plugins | grep nomad Handling requests# The ServeDNS function is used to handle the DNS request by the plugin. It takes a context.Context and a dns.ResponseWriter as arguments. The dns.ResponseWriter is used to write the response back to the client. The ServeDNS function returns an int which is the status code of the response. The status code is used by the next plugin in the chain to determine if it should handle the request or not. Since the nomad plugin expects a query in format of service.namespace.nomad, it validates the query and extracts the service name and namespace from it. If the query is invalid, it returns dns.RcodeServerFailure status code. If the query is valid, it queries the Nomad API for the service and returns the response. func (n Nomad) ServeDNS(ctx context.Context, w dns.ResponseWriter, r *dns.Msg) (int, error) { state := request.Request{W: w, Req: r} qname := state.Name() qtype := state.QType() // Split the query name with a `.` as the delimiter and extract namespace and service name. // If the query is not for a Nomad service, return. qnameSplit := dns.SplitDomainName(qname) if len(qnameSplit) < 3 || qnameSplit[2] != "nomad" { return plugin.NextOrFailure(n.Name(), n.Next, ctx, w, r) } namespace := qnameSplit[1] serviceName := qnameSplit[0] ... The plugin handles A,AAAA and SRV record requests currently. Since A/AAAA records can only contain an IP address, SRV records can be used to advertise the port number. // Check the query type to format the appriopriate response. switch qtype { case dns.TypeA: m.Answer = append(m.Answer, &dns.A{ Hdr: header, A: addr, }) case dns.TypeAAAA: m.Answer = append(m.Answer, &dns.AAAA{ Hdr: header, AAAA: addr, }) ... Caching# While some coredns plugins have an in-built support for caching the records to avoid a lookup to Nomad server everytime (which can get expensive), I decided to skip the caching implementation. This is because coredns itself has a cache plugins which supports a lot of various options for controlling the cache. In my testing, just using this cache plugin was sufficient to avoid Nomad lookups each time a query came in. Testing the plugin# I created a fake HTTP test server and added the URI paths which the Nomad Go client uses to query the Nomad API. This way I could test the plugin without having to run a Nomad cluster locally. // Setup a fake Nomad server. nomadServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { default: t.Errorf("Not implemented: %v", r.URL.Path) return case "/v1/service/example": w.Write([]byte(`[{"Address":"1.2.3.4","Namespace":"default","Port":23202,"ServiceName":"example"}]`)) case "/v1/service/fakeipv6": w.Write([]byte(`[{"Address":"1:2:3::4","Namespace":"default","Port":8000,"ServiceName":"fakeipv6"}]`)) case "/v1/service/multi": w.Write([]byte(`[{"Address":"1.2.3.4","Namespace":"default","Port":25395,"ServiceName":"multi"},{"Address":"1.2.3.5","Namespace":"default","Port":20888,"ServiceName":"multi"},{"Address":"1.2.3.6","Namespace":"default","Port":26292,"ServiceName":"multi"}]`)) case "/v1/service/nonexistent": w.Write([]byte(`[]`)) } })) Usage Example# Here are some examples of how this plugin works. The Corefile I’ve used is: nomad:1053 { errors debug health log nomad { address http://127.0.0.1:4646 ttl 10 } prometheus :9153 cache 30 } On running coredns, it connects to a local Nomad agent which is running at http://127.0.0.1:4646. I’m running a redis job in Nomad, so I can query the service using the following command: nomad service info -namespace=default redis Job ID Address Tags Node ID Alloc ID redis 192.168.29.76:25395 [] 9e02c85b 95170495 redis 192.168.29.76:20888 [] 9e02c85b a1cf923c redis 192.168.29.76:26292 [] 9e02c85b a9d1181a Now, the same query can also be handled using the DNS server running by coredns: doggo redis.default.nomad @tcp://127.0.0.1:1053 NAME TYPE CLASS TTL ADDRESS NAMESERVER redis.default.nomad. A IN 10s 192.168.29.76 127.0.0.1:1053 redis.default.nomad. A IN 10s 192.168.29.76 127.0.0.1:1053 redis.default.nomad. A IN 10s 192.168.29.76 127.0.0.1:1053 Quering an SRV record is also possible: dig +noall +answer +additional redis.default.nomad @127.0.0.1 -p 1053 SRV redis.default.nomad. 10 IN SRV 10 10 25395 redis.default.nomad. redis.default.nomad. 10 IN SRV 10 10 20888 redis.default.nomad. redis.default.nomad. 10 IN SRV 10 10 26292 redis.default.nomad. redis.default.nomad. 10 IN A 192.168.29.76 redis.default.nomad. 10 IN A 192.168.29.76 redis.default.nomad. 10 IN A 192.168.29.76 Code# You can checkout the source code here. Fin!]]> CoreDNS is an extensible DNS server (which is actually a fork of Caddy v1) that can be used to serve DNS records for a domain. It is written in Go and is very easy to extend. It has a plugin system that allows you to write your own plugins to extend its functionality. In this post, I will be writing a plugin for CoreDNS that will allow it to serve DNS records for Nomad services.

I recently came across a niche use case which required me to use a resolver address for querying Nomad services. Currently Nomad native service discovery is only possible via consul-template (which renders static block of address/port of services) and HTTP API. I felt adding a DNS interface would be a nice add-on.

Rather than writing and implementing all the boring crux of a DNS server, it’s better to extend on an existing server. CoreDNS fits in well here, it’s also used by K8s for service discovery. CoreDNS has an extensible plugin system which allows you to chain multiple different plugins for handling a request. Stuff like logs/metrics/caching comes for free with CoreDNS in form of a plugin.

CoreDNS docs have a handy guide on how to write a plugin from scratch, so I won’t cover that again here. These are just my short notes on how I locally developed the plugin, tested and some problems I encountered during the process.

Developing the plugin#

Firstly, you need to clone CoreDNS repo. Then, using the example plugin provided, you can create a new repository for your own plugin.

To make CoreDNS aware about this plugin, you need to add it to the plugin.cfg file. This file is used by the build script to generate the plugin list. The order of plugins matter here as they define how the request is handled. For example, if you want to log all the requests, you need to add the log plugin before your plugin.

To add an external plugin this is the format used:

nomad:github.com/mr-karan/coredns-nomad

However, since we are developing the plugin locally, we need to add a replace directive in go.mod file to point to the local plugin directory.

replace github.com/mr-karan/coredns-nomad => ../../coredns-nomad

Next, you can run make in coredns repository. It’ll build the binary and place it in coredns directory. You can run this binary to test your plugin. To check if the plugin indeed exists in the binary, you can use the following command

./coredns -plugins | grep nomad

Handling requests#

The ServeDNS function is used to handle the DNS request by the plugin. It takes a context.Context and a dns.ResponseWriter as arguments. The dns.ResponseWriter is used to write the response back to the client. The ServeDNS function returns an int which is the status code of the response. The status code is used by the next plugin in the chain to determine if it should handle the request or not.

Since the nomad plugin expects a query in format of service.namespace.nomad, it validates the query and extracts the service name and namespace from it. If the query is invalid, it returns dns.RcodeServerFailure status code. If the query is valid, it queries the Nomad API for the service and returns the response.

func (n Nomad) ServeDNS(ctx context.Context, w dns.ResponseWriter, r *dns.Msg) (int, error) {
	state := request.Request{W: w, Req: r}
	qname := state.Name()
	qtype := state.QType()

	// Split the query name with a `.` as the delimiter and extract namespace and service name.
	// If the query is not for a Nomad service, return.
	qnameSplit := dns.SplitDomainName(qname)
	if len(qnameSplit) < 3 || qnameSplit[2] != "nomad" {
		return plugin.NextOrFailure(n.Name(), n.Next, ctx, w, r)
	}
	namespace := qnameSplit[1]
	serviceName := qnameSplit[0]

...

The plugin handles A,AAAA and SRV record requests currently. Since A/AAAA records can only contain an IP address, SRV records can be used to advertise the port number.

		// Check the query type to format the appriopriate response.
		switch qtype {
		case dns.TypeA:
			m.Answer = append(m.Answer, &dns.A{
				Hdr: header,
				A:   addr,
			})
		case dns.TypeAAAA:
			m.Answer = append(m.Answer, &dns.AAAA{
				Hdr:  header,
				AAAA: addr,
			})

...

Caching#

While some coredns plugins have an in-built support for caching the records to avoid a lookup to Nomad server everytime (which can get expensive), I decided to skip the caching implementation. This is because coredns itself has a cache plugins which supports a lot of various options for controlling the cache. In my testing, just using this cache plugin was sufficient to avoid Nomad lookups each time a query came in.

Testing the plugin#

I created a fake HTTP test server and added the URI paths which the Nomad Go client uses to query the Nomad API. This way I could test the plugin without having to run a Nomad cluster locally.

	// Setup a fake Nomad server.
	nomadServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		switch r.URL.Path {
		default:
			t.Errorf("Not implemented: %v", r.URL.Path)
			return
		case "/v1/service/example":
			w.Write([]byte(`[{"Address":"1.2.3.4","Namespace":"default","Port":23202,"ServiceName":"example"}]`))
		case "/v1/service/fakeipv6":
			w.Write([]byte(`[{"Address":"1:2:3::4","Namespace":"default","Port":8000,"ServiceName":"fakeipv6"}]`))
		case "/v1/service/multi":
			w.Write([]byte(`[{"Address":"1.2.3.4","Namespace":"default","Port":25395,"ServiceName":"multi"},{"Address":"1.2.3.5","Namespace":"default","Port":20888,"ServiceName":"multi"},{"Address":"1.2.3.6","Namespace":"default","Port":26292,"ServiceName":"multi"}]`))
		case "/v1/service/nonexistent":
			w.Write([]byte(`[]`))
		}
	}))

Usage Example#

Here are some examples of how this plugin works. The Corefile I’ve used is:

nomad:1053 {
    errors
    debug
    health
    log
    nomad {
		address http://127.0.0.1:4646
        ttl 10
    }
    prometheus :9153
    cache 30
}

On running coredns, it connects to a local Nomad agent which is running at http://127.0.0.1:4646. I’m running a redis job in Nomad, so I can query the service using the following command:

nomad service info -namespace=default redis                 
Job ID  Address              Tags  Node ID   Alloc ID
redis   192.168.29.76:25395  []    9e02c85b  95170495
redis   192.168.29.76:20888  []    9e02c85b  a1cf923c
redis   192.168.29.76:26292  []    9e02c85b  a9d1181a

Now, the same query can also be handled using the DNS server running by coredns:

doggo redis.default.nomad @tcp://127.0.0.1:1053
NAME                	TYPE	CLASS	TTL	ADDRESS      	NAMESERVER     
redis.default.nomad.	A   	IN   	10s	192.168.29.76	127.0.0.1:1053	
redis.default.nomad.	A   	IN   	10s	192.168.29.76	127.0.0.1:1053	
redis.default.nomad.	A   	IN   	10s	192.168.29.76	127.0.0.1:1053

Quering an SRV record is also possible:

dig +noall +answer +additional redis.default.nomad @127.0.0.1 -p 1053 SRV
redis.default.nomad.	10	IN	SRV	10 10 25395 redis.default.nomad.
redis.default.nomad.	10	IN	SRV	10 10 20888 redis.default.nomad.
redis.default.nomad.	10	IN	SRV	10 10 26292 redis.default.nomad.
redis.default.nomad.	10	IN	A	192.168.29.76
redis.default.nomad.	10	IN	A	192.168.29.76
redis.default.nomad.	10	IN	A	192.168.29.76

Code#

You can checkout the source code here.

Fin!

]]> Karan Sharma https://ravidwivedi.in/posts/tough-day-in-albania/ https://ravidwivedi.in/posts/tough-day-in-albania/ Mon, 26 Dec 2022 21:16:03 GMT After attending DebConf22 in Kosovo, Akshat and I planned to spend a couple of days in Shkodër, Albania. From Prizren, Kosovo, we reached Tirana, Albania and stayed for a night. Many of the fellow Indians who attended DebConf22 were also with me. The next morning, we checked out and took a bus for Shkodër.

While Akshat boarded the bus and sat with Abraham, I decided to search for some snacks outside before boarding. As a consequence, the bus got full and I didn’t get a seat. Seeing this, a lady from Germany sitting beside Abraham made space for me. Therefore, three people were sitting on two seats. It was a nice gesture, but obviously uncomfortable.

The weather was scorchingly hot, and the bus fans weren’t working properly. After a two-hour ride, we deboarded in Shkodër. Akshat and I had booked an apartment from Airbnb. Going by the location, we figured out it was 1 kilometer away, which we decided to walk.

Upon reaching the location, we texted our apartment owner on Airbnb. On the other side was a lady, and we were not able to understand what she was saying as her texts weren’t in English. We tried translating her texts, but the translation didn’t make sense to us. She seemed to be saying that she has an appointment in the church at 3 o’clock.

It was very frustrating. We were standing in hot sun with our apartment nowhere in sight. Then we tried calling her as I had a local SIM card. A boy was passing the street we were on, and we handed the phone to him. However, he could not translate what the lady said, as he probably didn’t know English.

Then I left Akshat to search for anyone else who could translate. I came across a car repair shop where someone agreed to translate the call for me. After talking with the lady over the phone, the person told me that the location is 2 kilometers away and offered a ride in his car. I told him that I would need to bring my friend as well.

I went back to Akshat. A girl was passing by on a bicycle. We handed over the phone to her and phoned the lady from our Airbnb booking. The girl translated the call for us in English. She told us that they are calling us at Hotel Rozafa. We asked her for directions to Hotel Rozafa, and we later discovered that it was where the bus from Tirana dropped us, but we didn’t know at that time. We walked towards Hotel Rozafa and came across the car repair shop I mentioned above. Turned out that the person who offered help earlier had left the car repair shop by then.

Then a person at that car repair place gave us a few grapes and water. At that moment, a few grapes felt refreshing and energizing. Then we went ahead and bought water from a shop. The shopkeeper asked whether we were from Afghanistan, to which we said, “No, we are from India.”

Upon reaching a shop named Neptune, Akshat texted the lady again, telling her that we had reached Neptune. Turned out they knew that shop as a few minutes later, her husband was there. He came there by taxi to pick us up. It was at this point that their plan was revealed. Earlier, I wondered why they were calling us to Hotel Rozafa - it was to pick us up.

Finally, we reached our apartment and the lady and her husband gave instructions about how things work at the apartment. She had to call her sister in Italy every time she needed to talk to us as she didn’t know English, so her sister acted as a translator.

I have noticed with a couple of Airbnbs we had booked in Albania that the locations given online were not correct, and we had to ask for help from the locals. However, the locals in Albania were very helpful. Whether they knew English or not, they always tried to lend a helping hand. I think that the apartment owner taking a taxi to pick us up was a nice gesture as well.

]]> Ravi Dwivedi https://mrkaran.dev/posts/barreldb/ https://mrkaran.dev/posts/barreldb/ Sat, 17 Dec 2022 18:30:00 GMT int64(r.Header.Expiry). Redis Server# In addition to using BarrelDB as a Go library, I also implemented a redis-compatible server. I found tidwall/redcon as an easy-to-use library to create a Redis-compatible server for Go applications. All I’d do was wrap BarrelDB API methods and define handlers for SET / GET. I was able to use redis-cli and connect to the BarrelDB server: 127.0.0.1:6379> set hello world OK 127.0.0.1:6379> get hello "world" Benchmarks# You can check the repo for the actual benchmarks. However, I’d like to point out some inferences of the results from redis-benchmark. First, let’s send 100000 requests to the server using 50 parallel clients. This command creates a unique key for each SET operation. redis-benchmark -p 6379 -c 50 -t set -n 100000 -r 100000000 Summary: throughput summary: 145985.41 requests per second latency summary (msec): avg min p50 p95 p99 max 0.179 0.016 0.183 0.207 0.399 1.727 So, 140k requests per second is not bad at all for a disk-based KV. But the exciting thing to note here is that the performance is predictable even if you increase the load by increasing clients: redis-benchmark -p 6379 -c 200 -t set -n 100000 -r 100000000 Summary: throughput summary: 140845.08 requests per second latency summary (msec): avg min p50 p95 p99 max 0.718 0.224 0.711 0.927 1.183 5.775 If we increase the number of requests (by 5x) as well, the throughput looks almost the same: redis-benchmark -p 6379 -c 200 -t set -n 500000 -r 100000000 Summary: throughput summary: 138350.86 requests per second latency summary (msec): avg min p50 p95 p99 max 0.748 0.056 0.711 0.879 1.135 63.135 This magic is all because of the way Bitcask uses Log structured hash table (just append-only records for writing data). Even with a lot of records, all it has to do is to write to the end of the file, which avoids any expensive I/O operations. Summary# Overall, I am happy with BarrelDB implementation as I covered everything described in the paper. This project had excellent learning outcomes for me. I spent a lot of time coming up with a design for structuring different components and their API methods and handling all the edge scenarios during the compaction process. Although, full credit to Bitcask as it kept its design so elegant and minimal yet achieved some significant numbers in the benchmarks. This is also a reminder that simple need not necessarily mean less powerful. I look forward to implementing a distributed KV store by adding support for multiple BarrelDB nodes connected via Raft. For now, gonna enjoy some chai and release this project to the WWW :) Fin!]]> I’d been mulling around reading a computer science paper and implementing a project based on it. Distributed systems, Networking and Databases are some of the things that fascinate me a lot. However, I had been looking to implement a more approachable project to avoid getting inundated initially. And I happened to chance upon the Bitcask paper through Avinash’s project: CaskDB.

After giving a quick read of this reasonably short paper, I decided to write a Golang implementation of the same, as it looked like an exciting project. If you’re interested in checking out the complete project, checkout BarrelDB.

---

Bitcask is a disk-based key-value storage engine designed for fast read and write operations. It is mainly in production use by Riak (which is a distributed database) as one of the storage engines. Bitcask under the hood has a straightforward yet clever design. It writes to the file in an append-only mode. This means that writes are performed only by appending to the end of the file, thus avoiding the need to perform any random disk I/O seek.

Let’s look at various components of Bitcask:

Format of the record#

• CRC: Stores the checksum of the value to ensure data consistency
• Timestamp: Timestamp in UNIX format, stored as int32.
• Expiry: If the record has an expiry defined, then the timestamp, in UNIX format, is stored as int32.
• Key Size: Size of the key in bytes
• Value Size: Size of the value in bytes
• Key
• Value

This additional metadata stored alongside the key/value is represented with a fixed-width header. Each field is represented as int32, so the total size of the header is 4*5 = 20 bytes. Here’s the code which encodes and decodes this record:

type Record struct {
    Header Header
    Key    string
    Value  []byte
}

// Header represents the fixed width fields present at the start of every record.
type Header struct {
    Checksum  uint32
    Timestamp uint32
    Expiry    uint32
    KeySize   uint32
    ValSize   uint32
}

// Encode takes a byte buffer, encodes the value of header and writes to the buffer.
func (h *Header) encode(buf *bytes.Buffer) error {
    return binary.Write(buf, binary.LittleEndian, h)
}

// Decode takes a record object decodes the binary value the buffer.
func (h *Header) decode(record []byte) error {
    return binary.Read(bytes.NewReader(record), binary.LittleEndian, h)
}

The record is encoded in the binary format before storing it on the disk.

Datafile#

A “datafile” (term used for the DB file on disk) is an append-only record of all the write operations. An instance of Bitcask can have several datafiles. However, there’s only one “active” datafile. In BarrelDB, a goroutine runs in the background at regular intervals to check if the size of the active DB file has crossed the threshold and then rotates the active file. It appends this DB file to the list of “stale” data files. All the new writes only happen to the “active” data file, and the stale files are merged as a part of the “compaction” process (described later in the post).

Here’s how a datafile is represented:

type DataFile struct {
    sync.RWMutex

    writer *os.File
    reader *os.File
    id     int

    offset int
}

It contains different handlers for writing and reading the file. The reason we have 2 file handlers instead of re-using the same one is that the writer is only opened in an “append-only” mode. In addition, since the active file can be rotated, the writer can be set to nil, ensuring no new writes ever happen on that file.

    writer, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
    if err != nil {
        return nil, fmt.Errorf("error opening file for writing db: %w", err)
    }

    // Create a reader for reading the db file.
    reader, err := os.Open(path)
    if err != nil {
        return nil, fmt.Errorf("error opening file for reading db: %w", err)
    }

KeyDir#

In addition to storing the file on disk, Bitcask also stores additional metadata, which defines how to retrieve the record. This hashtable is a map of keys with this metadata and is referred to as KeyDir. An important point to note here is that the value is never stored in this map. This makes it possible for Bitcask to handle datasets more significant than what the RAM can hold.

// KeyDir represents an in-memory hash for faster lookups of the key.
// Once the key is found in the map, the additional metadata, like the offset record
// and the file ID is used to extract the underlying record from the disk.
// Advantage is that this approach only requires a single disk seek of the db file
// since the position offset (in bytes) is already stored.
type KeyDir map[string]Meta

// Meta represents some additional properties for the given key.
// The actual value of the key is not stored in the in-memory hashtable.
type Meta struct {
    Timestamp  int
    RecordSize int
    RecordPos  int
    FileID     int
}

Here, RecordPos tells the record’s position offset (in bytes) in the entire file. Since the position of the record is stored in memory along with the key, the retrieval of the key doesn’t require more than a single disk seek. Bitcask achieves really low latency even with many keys in the database. A file system read-ahead cache also helps boost the performance and comes for free - no need to design a separate caching mechanism.

Compaction#

As we looked at previously, a datafile is simply an append-only sequence of writes. Any modification of the key is merely a new record appended to the datafile. KeyDir overwrites the entry of the key with the new metadata, which contains the new location of the record. Thus all reads will automatically return the updated value.

Deletes are handled similarly by writing a “tombstone” record for the key. When the user requests the key after it’s been deleted, BarrelDB can check whether that value equals the tombstone value and return an appropriate error.

As you would have guessed, our database will grow unbounded if we don’t perform any garbage cleanup. The datafiles need to be pruned for deleting expired/deleted records and merging all stale files in a single active file - to keep the number of opened files in check. All of these processes are together called “Compaction”.

Let’s take a look at how each of these compaction routine works under the hood:

Merge#

The merge process iterates over all the keys inside KeyDir and fetches their value. The value could come from a stale file as well. Once the new keys/values are updated, it writes them to a new active file. All the old file handlers are closed, and the stale files are deleted from the disk. The KeyDir is updated similarly since the new records live in a different position/file.

Hints File#

Bitcask paper describes a way of creating a “hints” file initially loaded in the database for faster startup time. This file is essential to bootstrap KeyDir after a cold startup. This avoids iterating over all data files and reading their values sequentially. In BarrelDB, gob encoding is used to dump the KeyDir map as a gob dump.

// generateHints encodes the contents of the in-memory hashtable
// as `gob` and writes the data to a hints file.
func (b *Barrel) generateHints() error {
    path := filepath.Join(b.opts.dir, HINTS_FILE)
    if err := b.keydir.Encode(path); err != nil {
        return err
    }

    return nil
}

During the startup, BarrelDB checks the presence of a .hints file, decodes this gob dump, and loads the data in KeyDir.

Removing expired keys#

A goroutine runs at a configurable interval to check if the value of the key has expired. If it has, it deletes the entry from KeyDir. During the following merge process, since this entry won’t be present in KeyDir, it’ll automatically be removed when the new datafile is created.

To check if the key has expired, a simple check, like comparing their timestamps in UNIX epoch format, is enough: time.Now().Unix() > int64(r.Header.Expiry).

---

Redis Server#

In addition to using BarrelDB as a Go library, I also implemented a redis-compatible server. I found tidwall/redcon as an easy-to-use library to create a Redis-compatible server for Go applications. All I’d do was wrap BarrelDB API methods and define handlers for SET / GET.

I was able to use redis-cli and connect to the BarrelDB server:

127.0.0.1:6379> set hello world
OK
127.0.0.1:6379> get hello
"world"

Benchmarks#

You can check the repo for the actual benchmarks. However, I’d like to point out some inferences of the results from redis-benchmark.

First, let’s send 100000 requests to the server using 50 parallel clients. This command creates a unique key for each SET operation.

redis-benchmark -p 6379 -c 50 -t set -n 100000 -r 100000000

Summary:
  throughput summary: 145985.41 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.179     0.016     0.183     0.207     0.399     1.727

So, 140k requests per second is not bad at all for a disk-based KV. But the exciting thing to note here is that the performance is predictable even if you increase the load by increasing clients:

redis-benchmark -p 6379 -c 200 -t set -n 100000 -r 100000000

Summary:
  throughput summary: 140845.08 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.718     0.224     0.711     0.927     1.183     5.775

If we increase the number of requests (by 5x) as well, the throughput looks almost the same:

redis-benchmark -p 6379 -c 200 -t set -n 500000 -r 100000000

Summary:
  throughput summary: 138350.86 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.748     0.056     0.711     0.879     1.135    63.135

This magic is all because of the way Bitcask uses Log structured hash table (just append-only records for writing data). Even with a lot of records, all it has to do is to write to the end of the file, which avoids any expensive I/O operations.

Summary#

Overall, I am happy with BarrelDB implementation as I covered everything described in the paper. This project had excellent learning outcomes for me. I spent a lot of time coming up with a design for structuring different components and their API methods and handling all the edge scenarios during the compaction process. Although, full credit to Bitcask as it kept its design so elegant and minimal yet achieved some significant numbers in the benchmarks. This is also a reminder that simple need not necessarily mean less powerful.

I look forward to implementing a distributed KV store by adding support for multiple BarrelDB nodes connected via Raft. For now, gonna enjoy some chai and release this project to the WWW :)

Fin!

]]> Karan Sharma https://ravidwivedi.in/posts/updated-gpg-keys/ https://ravidwivedi.in/posts/updated-gpg-keys/ Sat, 19 Nov 2022 10:31:36 GMT I updated my gpg keys because I had a 3072 bits RSA key and Debian requires 4096 bits RSA or elliptic curve. The key fingerprint is:

FF7D B951 7CE1 E19B 6EFE 695F E0E5 BAFD 3BBF 70B3

Check my gpg page for details.

So, if you are from Debian community and signing someone else’s keys which they will be using for the purposes of debian, make sure they are using a 4096 bit RSA key or an elliptic curve key.

]]> Ravi Dwivedi https://nadh.in/blog/web2-web3/ https://nadh.in/blog/web2-web3/ Tue, 15 Nov 2022 00:00:00 GMT It took me several seconds to parse the casual quip “But, aren’t you folks web2?”. I probed further and they continued—“Isn’t Zerodha^[1] web2? Why don’t you convert it to web3?”. For the next few minutes, I struggled to explain how technologies, processes, people, regulations, laws, industry, and the entire legal and societal foundation that underlie an organisation, no matter how imperfect, aren’t “web2”, and that they can’t just be converted to “web3”, whatever that meant. To my question as to why they called a whole bunch of things web2 and why they think it should be converted to web3, they didn’t have an answer. This interaction happened a few months ago with a young developer in their early twenties working for an American web3 startup. Since then, I have had a few more interactions including a few startup pitches, all eerily similar. All young people in their early twenties right out of college working on web3 things, all dismissive of everything non-web3 as web2. This conversation came up again this week in light of the new meltdowns unraveling^[2] in the crypto world.

]]> Kailash Nadh https://www.prashanthudupa.com/lifestyle-changes/ https://www.prashanthudupa.com/lifestyle-changes/ Mon, 31 Oct 2022 08:30:39 GMT Prashanth Udupa Fitness Philosophy https://ravidwivedi.in/posts/intro-to-fediverse/ https://ravidwivedi.in/posts/intro-to-fediverse/ Sat, 29 Oct 2022 09:14:56 GMT Finally, Elon Musk is the owner of Twitter. On the first day after the takeover, Twitter has already undergone some major changes. If you are wondering what the big deal is, here is a reminder that tech giants work for profit and control, rather than to make the world a better place. While I do not consider the previous CEO of Twitter to be good, the takeover by Musk may have made people more receptive to the ideas I am going to discuss.

This is a problem of centralization

Centralized services such as Twitter are vulnerable to such handovers. Even if the company is good today, it may get compromised in future due to change of ownership. The same thing happened with WhatsApp. After Facebook took over WhatsApp, it later introduced a policy to share data of WhatsApp users with Facebook.

Introducing Fediverse

Fediverse is NOT another company or a startup, rather a community made project.

Fediverse is a decentralized social network, where users using different services can follow, boost posts, and communicate with each other, similar to how users with accounts on different email providers can email each other. Different services on Fediverse have different policies, so you can choose a service which suits your needs. Anyone can self host their instance of Fediverse, or pay someone to host for them(just like any task in the world). There is no single company or person who owns Fediverse, rather different instances have different admins, usually running service in their free time, so don’t forget to send them a donation if you like the service. Fediverse users do not get tied to a particular service, as when they change their service providers, they can migrate their friends and followers, so they don’t need to start from scratch again.

Fediverse is a collection of different types of services. For example, Mastodon is like Twitter(according to use-case), Friendica is like Facebook, Pixelfed is for sharing images, similar to Instagram, and PeerTube is for sharing videos, a replacement for YouTube.

Each service on Fediverse is free software which means you can add features youself as well. For example, let’s say Mastodon does not have a feature and you would like to add it, then you have the ability to run your modififed version of Mastodon on your server as well. You can pay someone to add that feature too, or you can collaborate with others who like the same feature and colectively crowdfund to add that feature.

You can learn more about Fediverse here. Further, this is a good unofficial guide to get started. We need to fight for our freedom. You can take a step by deleting your Twitter account, or just trying out Fediverse and bring your friends on Fediverse to reduce your dependence on Twitter, Facebook and Instagram. I suggest major organizations (which usually have resources) to switch to Fediverse, which will raise awareness about it to their users. If they don’t feel like deleting their Twitter account, Fediverse can reduce their dependence on it and, hopefully, more users will join.

My profile on Fediverse is here. Feel free to connect.

For further reading on this topic, check out this blog post by Free Software Community of India.

]]> Ravi Dwivedi https://www.evalapply.org/posts/clojure-mars-rover/index.html https://www.evalapply.org/posts/clojure-mars-rover/index.html Wed, 19 Oct 2022 00:00:00 GMT Aditya Athalye clojure functional_programming software_design architecture https://ravidwivedi.in/posts/couldnt-attend-libreoffice-conference/ https://ravidwivedi.in/posts/couldnt-attend-libreoffice-conference/ Tue, 04 Oct 2022 13:59:56 GMT This year, The Document Foundation (TDF) invited me to attend the LibreOffice conference in Milan, Italy. In order to travel to Italy, I needed a visa.

To apply for the same, I had to book an appointment with VFS Global and submit the required documents. The conference was to be held from the 28th of September to the 1st of October 2022. I started checking for visa appointments on the 10th of August. However, I got no available slots in Delhi. I also checked the availability of appointments in other cities. I saw appointments were available in Kolkata, which is pretty far from where I live, and I didn’t want to spend that much time and money for the visa.

After looking up for appointments at the VFS Delhi center for 8 consecutive days, I got one for the 22nd of August. During my appointment, I was informed by the staff that I had applied in the wrong category - tourist. They told me further that if my purpose of travel is to attend a conference, then the category of application should be “business” or “invitation.” Therefore, I was advised to book another appointment in the correct category. It was not a visa refusal. I didn’t get to pay for my visa fee or submit my documents.

I tried to book another appointment in the correct category meant for attending conferences, but didn’t get any slots for days, and so I finally gave up in mid-September and told the conference organizers that I wouldn’t be able to join.

Finally, I would like to end this post by thanking TDF for inviting me to the conference and offering to sponsor my costs. It is unfortunate that I could not attend.

]]> Ravi Dwivedi https://www.learningfromdata.zingg.ai/p/zingg-turns-one https://www.learningfromdata.zingg.ai/p/zingg-turns-one Sun, 02 Oct 2022 12:27:18 GMT I did not realize that a year has gone by working publicly on Zingg. But now that the calendar has made the fact sink in, it feels surreal. Wasn’t it just yesterday that I was fretting about the documentation before announcing the open source and penning down my reasons for building Zingg? Now here I am, recruiting the team for the enterprise version :-)

It feels great to complete an entire year! Days have passed fairly quickly, spread between writing, lots of reading, speaking, giving demos, and meeting people. We have done a lot of development work as well as spent time on incorporation and funding. I have thoroughly enjoyed interacting with users and learning about the different use cases enabled by identity resolution. We have more than 200 members on our Slack now, with representation from Fortune 500s, digital natives, data platforms, consulting companies and startups. Besides the Customer 360, personalisation, AML, KYC, GDPR, master data management use cases, the trend towards composable customer data platforms is fuelling a lot of interest in Zingg which is very promising. The data stack has proven patterns for data ingestion, transformation and activation, but identity is a core missing piece there. Zingg fits in very well into this space.

Thanks for reading Sonal’s Newsletter! Subscribe for free to receive new posts and support my work.

One of the fundamental issues with identity resolution is scalability. While building Zingg, we used example datasets and tested our algorithms on them. To test scalability, we copy pasted an example multiple times to build a dummy dataset of roughly 15 million rows. Most enterprise master data is sub 10m rows. Zingg scaled well on this. However, dummy datasets with the same rows perform differently from actual data with real distribution of values. So there were some lingering doubts in the back of my mind. These were soon resolved when we consulted with a Fortune 10 having 12.5m records where Zingg ran like a breeze. Hence, I was confident of scalability while releasing the open source, but I wasn’t sure how far it would go beyond 15m records without tuning the knobs. Hence, it is extremely uplifting to see our users resolve identities on 70-80m records in a few hours with Zingg. Due to the quadratic nature of the comparisons, this is at least 25 times more complex than our 15m endeavours, and seeing it work well is a great confidence boost! Another measure for Zingg is the accuracy of predictions, and users have done head to head comparisons with their multi-million dollar MDM tools to confirm our results. Yay!

One very important aspect for us is arming our users with the power of machine learning and distributed systems in a self-serve, no-code way. Many of our users are first timers with Java, Spark or ML. Zingg abstracts away the technical complexity, and gives the superpower of identity resolution at scale directly to the end user. Now that our Python release is out, we are actively working on a native Snowflake implementation. This opens up a new way for Snowflake users to use Zingg and identify their customers and personalize their journeys and experiences.

There have been some mistakes last year, and I plan to do better in the coming year. Due to my flow state, I missed applying to some conferences but the one I am most disappointed about missing is Coalesce from dbt Labs. I would have met some amazing data practitioners there and made many friends. This time I will clear my calendar and attend virtually. If you are joining, do let me know.

There have been some other disappointments, but they were in the making since some time. Can’t complain.

On a philosophical note, identity resolution is the discovery of self as well as building relationships. Working on Zingg enables me to learn a lot of things around communication, positioning, customer needs, technology, hiring and sales. It pushes my boundaries and makes me discover myself. It also helps me talk to a lot of people and build relationships with customers, data leaders, practitioners, evangelists, contributors, team, investors and mentors. Many of us feel lost at the thought of networking - I am not sure how I would do if I had the charter to meet x new people every week! But when the baseline is common ground on data or ML, it is much easier to connect.

Zingg is made with meraki. It has been a lot of hard work, but it doesn’t feel that way at all. Last year has been nice, and we are having a good time. This coming year will decide how things shape up for us as a project and a startup.

Thanks to all of you for investing your time and effort on Zingg. We are incredibly grateful and motivated to do even more this year.

Wish us luck!

Thanks for reading Sonal’s Newsletter! Subscribe for free to receive new posts and support my work.

]]> Sonal Goyal https://mrkaran.dev/posts/nomad-logging/ https://mrkaran.dev/posts/nomad-logging/ Sat, 03 Sep 2022 18:40:55 GMT 8888 On sending an HTTP request using cURL we can see the logs that this webserver generated: curl -i 192.168.29.76:31775 HTTP/1.0 200 OK Server: SimpleHTTP/0.6 Python/3.10.4 Date: Sun, 04 Sep 2022 06:18:45 GMT Content-type: text/html; charset=utf-8 Content-Length: 869 ... Nomad stores the logs inside the applications’ allocation directory, inside /opt/nomad/data/alloc. To see the logs for the above allocation ID, we can use: tail -f /opt/nomad/data/alloc/1d05d64b-4b59-3c65-8261-760499d9e4f6/alloc/logs/server.stderr.0tail -f server.stderr.0 192.168.29.76 - - [04/Sep/2022 11:48:26] "GET / HTTP/1.1" 200 - 192.168.29.76 - - [04/Sep/2022 11:48:45] "GET / HTTP/1.1" 200 - Enriching logs with metadata# As you can see, these logs are precisely what the python3 -m http.server command generates. Ideally, Nomad should have enriched these logs with metadata about the allocation ID, job name, namespace, the node it’s running on, etc., as noted in this GitHub issue. However, since that’s not yet available, I brainstormed a few different options: Approach 1: Sidecar# The first approach was to run vector as a sidecar next to the main task. This is the simplest option, to begin with. Vector can be independently configured to add metadata for the logs collected from the allocation directory of the group. However, as with every sidecar deployment, there’s a lot of extra resource usage. For every 10 different groups, reserving resources for 10 vector agents quickly eats up your available CPU/Memory of the underlying node. A more critical downside, though, was asking every developer to also configure a Vector sidecar job. And to keep all these configs in sync to ensure they’re unified across namespaces is also another headache. Due to these reasons, I discarded this option early on. However, suppose your deployment scale (managing applications) is relatively smaller. In that case, this is actually not a bad idea. Approach 2: Events Stream# My next option was to listen to the Nomad events stream and generate a “vector” configuration template to collect logs and enrich them with metadata from the Events Stream. I developed v0.1 of nomad-vector-logger based on this concept. Since I’ve written a wrapper to collect events from Nomad using nomad-events-sink it was relatively straightforward to extend it to generate a vector.toml config. However, after testing in prod for a few days, I noticed that relying on the events stream is unreliable. Nomad events are not WebSocket based (as of yet). It’s a simple long polling mechanism which sends events to a Go channel as and when they happen. What happens when you miss an event? What happens when you run nomad system gc, which clears the events index? These were some of the challenges I faced with this v0.1 approach. There needs to be some sort of “reconcile” mechanism that periodically runs. A reconciliation loop that lists all allocations using the HTTP API can help whenever there are missing events. Approach 3: Enrichment Tables# I also posted about the above program in Vector’s discord group (they’re super active+helpful folks) and discussed this daemon with them. They also suggested a simpler alternative: generating a CSV of running allocations instead of a .toml config. Vector has support for Enrichment Tables which means that it can “lookup” a CSV file to find a particular row and enrich the log event with the information found from the CSV. This seemed a super cool idea, and I developed v0.2 using this. Super thankful to Vector maintainers for giving me this idea! However, this approach had a few “subtle” drawbacks that I found: vector doesn’t support live-reloading if the CSV file changes. vector has support for watching a config file for changes or sending a SIGHUP to reload. However, that only works for vector’s own config files. Since the CSV file is an external file, vector cannot watch it for changes. I came up with an ugly bash script hack and compared the md5 hash of the file in a while loop and if it changed, then send a SIGHUP to vector. All I can say is it’s ugly, but it works. If you wish to see it, it’s available here in all it’s glory. The most significant issue was the chance of losing logs for the initial 10-20s of a new allocation. The above shell script had a sleep(10) because md5sum can be a bit CPU intensive to keep frequently doing. Vector sees a new allocation and starts ingesting events. It tries to look up the CSV row by the allocation ID, but it doesn’t find it yet in the CSV file, complains about it, and drops the log event. Thus, I had to drop the CSV idea in search to find another more reliable approach to this. For people interested in this approach, you can checkout the csv branch here. Approach 4: Periodic Reconciliation Loop# The final v0.3.0 solution, which IMHO fixed all the above issues, was: Skip Nomad events stream. Since I have to build a reconciliation loop anyway, listening to events is just extra work without tangible benefits. I used a background Goroutine channel to periodically refresh the list of allocations running on that node. Even if I fetched this list once every 30s or so, it’s OK because Vector will start ingesting logs once the config gets generated. It will start reading the file from the beginning. So logs aren’t lost even if I templated the config much later after the alloc began running. I added the support to delay the removal of allocation from the file. If an allocation is stopped (e.g., a new version is deployed or the job restarted), the program doesn’t immediately removes the allocation from the config file. The user can set a delay period which works like a cooling down period. In this period, one can assume that Vector would have finished sending all logs to the upstream. In case the application generates too many logs faster than what the upstream sink can accept (e.g. if the upstream Elasticsearch gets slower). Suppose we remove the allocation _immediately whenever it stops. In that case, there’s a probability that Vector wouldn’t have read the file to the end. This cooling period helps to ensure that doesn’t happen. This is not a fool-proof situation but should cover most cases unless the upstream sink is dead for many hours. How it works# Now that we’ve covered a few different approaches and the pros/cons of each let’s see how nomad-vector-logger works. Essentially nomad-vector-logger is meant to be deployed inside a Nomad cluster as a system job. A system job in Nomad runs on each node. Whenever a new node gets added to the cluster, Nomad’s scheduler schedules a copy of this program on that new node automatically. This is the equivalent of a “Daemonset” in K8s. nomad-vector-logger uses Nomad’s HTTP API to query all nodes’ running allocations. Once it gets the list, it adds it to an internal map and signals to generate a config. The final config that is templated out looks like this: [sources.source_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy] type = "file" include = [ "/opt/nomad/data/alloc/64a2f9fd-e003-0bb3-b5cd-838125283a06/alloc/logs/proxy*" ] line_delimiter = "\n" read_from = "beginning" [transforms.transform_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy] type = "remap" inputs = ["source_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy"] source = ''' # Store Nomad metadata. .nomad.namespace = "default" .nomad.node_name = "pop-os" .nomad.job_name = "nginx" .nomad.group_name = "nginx" .nomad.task_name = "proxy" .nomad.alloc_id = "64a2f9fd-e003-0bb3-b5cd-838125283a06" ''' For people unfamiliar with vector, it’s essentially doing 2 things: Get logs from a “file” source. The file path comes from nomad-vector-logger (where all the logs for proxy task are located) It adds a JSON object nomad with relevant keys. Vector pipeline will send this event to another “transformer” which can further process the log event (for eg parsing it as logfmt or JSON etc) and then finally send it to an upstream sink like Loki/Elasticsearch etc. Here’s an example of the before/after of a log line shown above in this post: Before# After# Perfect! We’ve annotated the same log event with Nomad metadata, and Vector will be able to identify these logs. If you’re interested in a complete setup on deploying this to Nomad, take a look at dev setup which contains a Nomad jobspec to deploy nomad-vector-logger as a sidecar with vector as the main task. Hope this post helped you start configuring a logging pipeline for applications running with non-docker task drivers. Fin!]]> Application orchestrators like Nomad, Kubernetes etc., allow you to deploy multiple applications on the same host. Logs are stored on the underlying node wherever the applications are run. However, it’s pretty common to treat these instances as ephemeral if they’re a part of an autoscaling group. Therefore depending on the node’s availability to search these logs is not a practical idea as the node can go down anytime. Moreover, in most cases, access to these nodes is limited to cluster administrators, not the maintainers (developers/application owners). In such cases, a log shipping agent must ship and aggregate logs from all cluster nodes and store them centrally (like Elasticsearch, Clickhouse, Loki).

I’d recommend reading this excellent post by Adrian, who has explained how to set up a Vector logging pipeline for applications running with docker task driver and ship the logs to Loki. For the applications running using docker task driver, Nomad piggybacks to docker daemon for configuring logging options. Docker daemon supports many logging options; in my experience, the journald log driver works reliably well.

However, this post is about tasks not using docker but any other driver (e.g. raw_exec and exec). Nomad doesn’t provide many configuration options for logging for these drivers. The biggest issue is that Nomad logs the application’s stdio/stderr stream to the log directory as-is without annotating any metadata about the task. This means that if you’ve multiple applications running on one host, the log shipping agent will not be able to identify which application’s logs are being ingested.

Consider this as an example. We’re running a simple web server using exec driver:

job "http" {
  datacenters = ["dc1"]
  type        = "service"

  group "app" {
    count = 1
    network {
      mode = "bridge"
      port "python-http" {
        to = "8888"
      }
    }

    task "server" {
      driver = "exec"

      config {
        command = "/usr/bin/python3"
        args    = ["-m", "http.server", "8888"]
      }
    }
  }
}

Once the alloc is running, we can find its IP address using:

nomad alloc status 1d05d64b | grep -A 3 'Allocation Addresses'
Allocation Addresses (mode = "bridge")
Label         Dynamic  Address
*python-http  yes      192.168.29.76:31775 -> 8888

On sending an HTTP request using cURL we can see the logs that this webserver generated:

curl -i 192.168.29.76:31775
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.10.4
Date: Sun, 04 Sep 2022 06:18:45 GMT
Content-type: text/html; charset=utf-8
Content-Length: 869
...

Nomad stores the logs inside the applications’ allocation directory, inside /opt/nomad/data/alloc. To see the logs for the above allocation ID, we can use:

tail -f /opt/nomad/data/alloc/1d05d64b-4b59-3c65-8261-760499d9e4f6/alloc/logs/server.stderr.0tail -f server.stderr.0 
192.168.29.76 - - [04/Sep/2022 11:48:26] "GET / HTTP/1.1" 200 -
192.168.29.76 - - [04/Sep/2022 11:48:45] "GET / HTTP/1.1" 200 -

Enriching logs with metadata#

As you can see, these logs are precisely what the python3 -m http.server command generates. Ideally, Nomad should have enriched these logs with metadata about the allocation ID, job name, namespace, the node it’s running on, etc., as noted in this GitHub issue.

However, since that’s not yet available, I brainstormed a few different options:

Approach 1: Sidecar#

The first approach was to run vector as a sidecar next to the main task. This is the simplest option, to begin with. Vector can be independently configured to add metadata for the logs collected from the allocation directory of the group. However, as with every sidecar deployment, there’s a lot of extra resource usage. For every 10 different groups, reserving resources for 10 vector agents quickly eats up your available CPU/Memory of the underlying node. A more critical downside, though, was asking every developer to also configure a Vector sidecar job. And to keep all these configs in sync to ensure they’re unified across namespaces is also another headache. Due to these reasons, I discarded this option early on. However, suppose your deployment scale (managing applications) is relatively smaller. In that case, this is actually not a bad idea.

Approach 2: Events Stream#

My next option was to listen to the Nomad events stream and generate a “vector” configuration template to collect logs and enrich them with metadata from the Events Stream. I developed v0.1 of nomad-vector-logger based on this concept. Since I’ve written a wrapper to collect events from Nomad using nomad-events-sink it was relatively straightforward to extend it to generate a vector.toml config. However, after testing in prod for a few days, I noticed that relying on the events stream is unreliable. Nomad events are not WebSocket based (as of yet). It’s a simple long polling mechanism which sends events to a Go channel as and when they happen. What happens when you miss an event? What happens when you run nomad system gc, which clears the events index? These were some of the challenges I faced with this v0.1 approach. There needs to be some sort of “reconcile” mechanism that periodically runs. A reconciliation loop that lists all allocations using the HTTP API can help whenever there are missing events.

Approach 3: Enrichment Tables#

I also posted about the above program in Vector’s discord group (they’re super active+helpful folks) and discussed this daemon with them. They also suggested a simpler alternative: generating a CSV of running allocations instead of a .toml config. Vector has support for Enrichment Tables which means that it can “lookup” a CSV file to find a particular row and enrich the log event with the information found from the CSV. This seemed a super cool idea, and I developed v0.2 using this. Super thankful to Vector maintainers for giving me this idea!

However, this approach had a few “subtle” drawbacks that I found:

• vector doesn’t support live-reloading if the CSV file changes. vector has support for watching a config file for changes or sending a SIGHUP to reload. However, that only works for vector’s own config files. Since the CSV file is an external file, vector cannot watch it for changes. I came up with an ugly bash script hack and compared the md5 hash of the file in a while loop and if it changed, then send a SIGHUP to vector. All I can say is it’s ugly, but it works. If you wish to see it, it’s available here in all it’s glory.
• The most significant issue was the chance of losing logs for the initial 10-20s of a new allocation. The above shell script had a sleep(10) because md5sum can be a bit CPU intensive to keep frequently doing. Vector sees a new allocation and starts ingesting events. It tries to look up the CSV row by the allocation ID, but it doesn’t find it yet in the CSV file, complains about it, and drops the log event. Thus, I had to drop the CSV idea in search to find another more reliable approach to this. For people interested in this approach, you can checkout the csv branch here.

Approach 4: Periodic Reconciliation Loop#

The final v0.3.0 solution, which IMHO fixed all the above issues, was:

• Skip Nomad events stream. Since I have to build a reconciliation loop anyway, listening to events is just extra work without tangible benefits.
• I used a background Goroutine channel to periodically refresh the list of allocations running on that node. Even if I fetched this list once every 30s or so, it’s OK because Vector will start ingesting logs once the config gets generated. It will start reading the file from the beginning. So logs aren’t lost even if I templated the config much later after the alloc began running.
• I added the support to delay the removal of allocation from the file. If an allocation is stopped (e.g., a new version is deployed or the job restarted), the program doesn’t immediately removes the allocation from the config file. The user can set a delay period which works like a cooling down period. In this period, one can assume that Vector would have finished sending all logs to the upstream. In case the application generates too many logs faster than what the upstream sink can accept (e.g. if the upstream Elasticsearch gets slower). Suppose we remove the allocation _immediately whenever it stops. In that case, there’s a probability that Vector wouldn’t have read the file to the end. This cooling period helps to ensure that doesn’t happen. This is not a fool-proof situation but should cover most cases unless the upstream sink is dead for many hours.

How it works#

Now that we’ve covered a few different approaches and the pros/cons of each let’s see how nomad-vector-logger works. Essentially nomad-vector-logger is meant to be deployed inside a Nomad cluster as a system job. A system job in Nomad runs on each node. Whenever a new node gets added to the cluster, Nomad’s scheduler schedules a copy of this program on that new node automatically. This is the equivalent of a “Daemonset” in K8s.

nomad-vector-logger uses Nomad’s HTTP API to query all nodes’ running allocations. Once it gets the list, it adds it to an internal map and signals to generate a config.

The final config that is templated out looks like this:

[sources.source_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy]
type = "file"
include = [ "/opt/nomad/data/alloc/64a2f9fd-e003-0bb3-b5cd-838125283a06/alloc/logs/proxy*" ]
line_delimiter = "\n"
read_from = "beginning"

[transforms.transform_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy]
type = "remap"
inputs = ["source_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy"]
source = '''
# Store Nomad metadata.
.nomad.namespace = "default"
.nomad.node_name = "pop-os"
.nomad.job_name = "nginx"
.nomad.group_name = "nginx"
.nomad.task_name = "proxy"
.nomad.alloc_id = "64a2f9fd-e003-0bb3-b5cd-838125283a06"
'''

For people unfamiliar with vector, it’s essentially doing 2 things:

• Get logs from a “file” source. The file path comes from nomad-vector-logger (where all the logs for proxy task are located)
• It adds a JSON object nomad with relevant keys.

Vector pipeline will send this event to another “transformer” which can further process the log event (for eg parsing it as logfmt or JSON etc) and then finally send it to an upstream sink like Loki/Elasticsearch etc.

Here’s an example of the before/after of a log line shown above in this post:

Before#

After#

Perfect! We’ve annotated the same log event with Nomad metadata, and Vector will be able to identify these logs. If you’re interested in a complete setup on deploying this to Nomad, take a look at dev setup which contains a Nomad jobspec to deploy nomad-vector-logger as a sidecar with vector as the main task.

Hope this post helped you start configuring a logging pipeline for applications running with non-docker task drivers.

Fin!

]]> Karan Sharma https://www.learningfromdata.zingg.ai/p/to-python-with-love https://www.learningfromdata.zingg.ai/p/to-python-with-love Fri, 26 Aug 2022 07:23:43 GMT There was a feeling of suspense and excitement hanging in the air. From the depths of the cubicles rose a bewitching hymn, whose clicks and clacks surrounded the room. Despite this alluring melody, silence and serenity prevailed. The countless fingers on the various keyboards pressed and released in sync as if they were playing one giant piano. Everyone typed in tandem and the entire scene felt straight out of an orchestra. Time, which stood eternally and unmovingly, slid when a delighted exclamation rang through the air. A gentle whoosh brought out the new Zingg feature, acting as a spell breaker to the enchantment everyone was under…

As you might have guessed by now, we just released a new version of Zingg! This release marks a complete shift from the command line training and matching jobs to a full programmatic interface of building applications with Zingg. With Python.

Thanks for reading Sonal’s Newsletter! Subscribe for free to receive new posts and support my work.

After Zingg became open source, we found most of our early users came from a non-Java, non-Spark background. Though it was extremely encouraging to see a broader user base for Zingg than originally anticipated, it had been a poor assumption at our end that everyone had the JVM running on their laptops to run Zingg. Hats off to the early adopters who still went ahead and resolved their entities with Zingg, we had definitely not made working with Zingg easy for them :-(

This realization led to our urgent Docker release, and it felt that we had crossed a big hurdle! Life and work moved on, and as interactions with users increased, one common pattern emerged - the use of Python. A lot of our users are writing Python data pipelines to get data from multiple sources. They are wrangling data in Python before putting it through Zingg. They are consuming the Zingg output results in Python based machine learning pipelines for risk modeling, customer segmentation, and product recommendations. They are on Databricks notebooks. Wouldn’t it be cool if such users could use Zingg through Python itself?

With this in mind, we set about defining a Python interface for Zingg. Our biggest challenge with Python was that the majority of our source code and all our core algorithms are in Java/Scala. Due to the computationally intensive nature of the identity resolution problem, we keep a very tight and performance-focused implementation. Rewriting everything would not make sense, though we did think about it for a fleeting moment. After some tension, we realized that Apache Spark was structured similarly, and realized we could borrow the best practices from there. This turned out to be a really good decision. In fact, we referred to the Apache Spark codebase a lot to build a Py4J-based interface to Zingg. We would invoke our Java code from Python, all the Java interfaces would be replicated on the Python side. Things started to look up again.

There was one more hurdle though. Something we realized much later as we got closer to shipping - Python packaging. Unlike a usual Python application, our interface is dependent on the compiled Java code and hence we had to figure out a way to include this as part of our package. Luckily, Spark is doing something similar so the packaging there became our big inspiration. Again!

With these major hurdles crossed, we got to the grind and were finally ready to put our work out. Honestly, this release has been a real big one - not only did we build the whole Python interface, but we have also added lots of cool things to core Zingg. Stopwords - stuff to ignore while matching attributes for example. Or special match types for emails or pin codes. More diagnostic messages while running Zingg - getting better there slowly and steadily!

In the end, once the release was done, it felt like a triumph! A mini celebration, and having a different version of fun - rest, recharge and laze around a bit. I haven’t heard any user complaints so far so it is probably working as planned ;-)

Among other things, I am very excited about the ability to finally have a good integration point with dbt. Besides Python, SQL through dbt is something we frequently see (of course! :-)). With dbt supporting Python, we have a cool new way for Zingg to work with dbt. I can’t wait to try this out….or maybe someone is working on this already! Are you The One?

Thanks for reading Sonal’s Newsletter! Subscribe for free to receive new posts and support my work.

]]> Sonal Goyal https://nadh.in/blog/triangle-of-fulfilment/ https://nadh.in/blog/triangle-of-fulfilment/ Mon, 22 Aug 2022 00:00:00 GMT “It’s 2022. Why hasn’t someone done it already!?”, I find exclaiming frequently when stumbling upon things and ideas that are relatively simple and so obvious that they should exist, but for some reason, don’t. It is frequently about software, occasionally about physical things, and once in a while, about an organisation that focuses on a certain cause that really ought to exist. It is of course not about hard problems like cold fusion or disease and poverty eradication, but things like … a mailing list manager.

]]> Kailash Nadh https://shrirangkahale.com/posts/termux-armbian/ https://shrirangkahale.com/posts/termux-armbian/ Tue, 16 Aug 2022 07:19:31 GMT Shrirang Kahale https://mrkaran.dev/posts/stop-which/ https://mrkaran.dev/posts/stop-which/ Sun, 07 Aug 2022 18:30:00 GMT /etc/alternatives/which $ ls -laht /etc/alternatives/which lrwxrwxrwx 1 root root 26 Apr 26 22:25 /etc/alternatives/which -> /usr/bin/which.debianutils Now, notice this: $ which ls ls: aliased to ls --color=tty $ /usr/bin/which.debianutils ls /usr/bin/ls Both are the same programs. However, why is the output different? This is because which is apparently a shell built-in zsh, that is why: # zsh which which which: shell built-in command # bash which which /usr/bin/which The inconsistency happens because zsh treats which as a shell built-in when it’s apparently not one. This article has some more details on why which is bad and how the Debian team is slowly deprecating it from being a part of debianutils anymore. When I invoked which from inside a Nomad chroot, it complained that I didn’t have /bin/sh (because I was using a custom chroot mount). I started looking hard for alternatives because this silly utility already wasted too much of my time. What to use# Use command -v. It’s a shell built-in, so it avoids a dependency on an external binary (unlike which). Usage example:# if ! command -v aws > /dev/null; then echo "Can't find 'aws' executable. Aborted." exit 1 fi References# I found the following posts while digging which and its alternatives. which-not-posix why-not-use-which-what-to-use-then Fin]]> which is a non-standard/non-POSIX compliant program. I faced many issues getting which to work in a chroot environment (Nomad).

So basically, which is a simple shell script program to find out the dependency by searching the $PATH (which is what makes it less deterministic). It’s also somehow symlinked 3 levels deep in Debian:

$ ls -laht /usr/bin/which
lrwxrwxrwx 1 root root 23 Apr 26 22:25 /usr/bin/which -> /etc/alternatives/which
$ ls -laht /etc/alternatives/which
lrwxrwxrwx 1 root root 26 Apr 26 22:25 /etc/alternatives/which -> /usr/bin/which.debianutils

Now, notice this:

$ which ls
ls: aliased to ls --color=tty

$ /usr/bin/which.debianutils ls
/usr/bin/ls

Both are the same programs. However, why is the output different? This is because which is apparently a shell built-in zsh, that is why:

# zsh
which which
which: shell built-in command

# bash
which which
/usr/bin/which

The inconsistency happens because zsh treats which as a shell built-in when it’s apparently not one.

This article has some more details on why which is bad and how the Debian team is slowly deprecating it from being a part of debianutils anymore.

When I invoked which from inside a Nomad chroot, it complained that I didn’t have /bin/sh (because I was using a custom chroot mount). I started looking hard for alternatives because this silly utility already wasted too much of my time.

What to use#

Use command -v. It’s a shell built-in, so it avoids a dependency on an external binary (unlike which).

Usage example:#

if ! command -v aws > /dev/null; then
        echo "Can't find 'aws' executable. Aborted."
        exit 1
fi

References#

I found the following posts while digging which and its alternatives.

• which-not-posix
• why-not-use-which-what-to-use-then

Fin

]]> Karan Sharma https://mrkaran.dev/posts/indiafoss-2022/ https://mrkaran.dev/posts/indiafoss-2022/ Sat, 06 Aug 2022 18:30:00 GMT On June 28, I received a text from Anand nudging me to submit a CFP for IndiaFOSS. Until then, I’d not planned to attend IndiaFOSS because travelling to participate in a conference didn’t seem enticing. However, the thought of visiting Bangalore (a city where I spent five years before Covid took the joy from our lives) stayed in my mind for the rest of the day. Call it serendipity or just that little external motivation by Anand; I decided to submit a proposal on Self Hosting with Nomad. IndiaFOSS would be the first physical conference I would attend after the pandemic. I love attending conferences because you get to meet and interact with many people (I somehow dislike the term “networking with people” because it’s heavily connoted with marketing/sales pitches).

IndiaFOSS is a v2 of a small pilot conference IndiaOS that happened in 2020. Around 650 people attended IndiaFOSS, and the execution scale was much bigger this time. IndiaFOSS is a 100% volunteer-driven conference, yet everything was quite professionally managed. My friend Dhiraj travelled from Chennai to attend the conference, and I hung out with him for most of the next two days.

Filling our tummies with a scrumptious breakfast of Thatte Idlis, we headed to attend the keynote by Rushabh, who gave an excellent overview of FOSS United and its journey so far. There were a series of exciting talks right after - especially by Kovid, a prolific FOSS contributor from India. Kovid’s software is widely used across the globe. He shared the story of how calibre evolved from a bunch of scripts written to get around Sony’s DRM to become the most sought-after ebook management tool that it is now. I’ve myself used calibre in one of my initial projects(back when I was still learning Python, please don’t judge that codebase). Next was the much-awaited interview of Rudra Saraswat - the 12-year-old whiz kid who’s a primary contributor to Ubuntu Unity Remix and many other projects. The entire auditorium was stunned to witness how this bright young kid manages his studies, extra-curricular and FOSS projects with a laser-sharp focus.

I spent some time reviewing my slides which were scheduled right after an excellent talk on hpw the Kubernetes release team works - by Nabarun. I was pleasantly surprised to see many people interested in Self Hosting/Nomad. You can check out the slides here.

It was time for lunch, and I finally got to meet Vinayak, Bibhas, Gaurav and Raghav with whom I’ve been interacting in a small Discord group since so many months! This is what pandemic years took away from us, sigh!

After lunch, Kailash presented his fantastic talk on languages, dictionaries, and his FOSS project - dictpress. Kailash was anticipating that people would be drowsy after lunch, but I doubt anyone would want to miss this talk. He gave a glimpse of the monumental work done by Datuk KJ Joseph and V. Krishna in the field of dictionaries and how dictpress makes it easier for anyone to host a dictionary website for any language.

The rest of the day was spent hanging out in the lobby, meeting fellow hackers, and chit-chatting. I consider myself an introvert, but I guess with developers, I tend to speak up openly because we tend to ignore any small-talks :P. Rainy evenings and a weekend meant all the Peak Bengaluru memes about not finding cabs were experienced first-hand.

Day 2 had quite a good lineup of talks as well. Krutika shared the designer’s dilemma of contributing to FOSS Projects. Nemo talked about how we could get a FOSS UPI app and touched upon the openness of UPI. We even did an open space around Self Hosting where many of us self-hosted enthusiasts came together and discussed different topics/problems we encountered while self-hosting. It was my first time interacting with Abhas, who runs Deeproot Linux. He shared his wisdom and experiences from self-hosting email servers, media servers and just about anything. I was pretty motivated by his dedication.

---

The best part about IndiaFOSS was that this wasn’t a conference meant just for techies. The talks highlighted every domain of FOSS- policy, design, languages, and hardware rather than just keeping it limited to software. I was happy to see a conference not peddled with any “corporate” product talks. This is what all FOSS conferences should look like, I guess?

I look forward to attending (and hopefully volunteering) next year’s conference!

Fin

]]> Karan Sharma https://nadh.in/blog/reflections-on-indiafoss-2022/ https://nadh.in/blog/reflections-on-indiafoss-2022/ Mon, 01 Aug 2022 00:00:00 GMT IndiaFOSS 2.0^[1], the second edition of the conference organised by the FOSS United Foundation, of which I am a part of, was held in Bengaluru on the 23rd and 24th of July. Previously named IndiaOS, the first edition that ran in January 2020 was an experiment that turned out to be a small, nice gathering of ~100 people in a small hall in a corner of JP Nagar, a residential area in South Bengaluru. There were talks, discussions, and food. It was nice. The conference was meant to be a yearly affair, which the COVID fiasco stalled, like a billion other human affairs.

]]> Kailash Nadh https://ravidwivedi.in/posts/layover-in-dubai/ https://ravidwivedi.in/posts/layover-in-dubai/ Sun, 31 Jul 2022 07:49:11 GMT A few days ago, I had a layover in Dubai while returning from a fun DebConf in Kosovo. The layover was a bit longer than 24 hours. I was joined by my friend Abraham Raji, who had to go to Kochi, and had a layover of around 22 hours.

Since my layover was longer than 24 hours, I needed a transit visa for boarding my Dubai to Delhi flight. For layovers less than 24 hours, the transit visa was not a requirement for boarding the next flight. Note that you just need to fill a form online to obtain a transit visa for UAE. Since my travel agent booked the flights, they took care of the transit visa on my behalf for a fee of 2,000 INR.

My flight from Tirana landed at the Terminal 2 of the Dubai airport at around 21:30 local time. Since I had a transit visa, my plan was to roam around in Dubai and return to the airport to catch my connecting flight. Therefore, I went through immigration. The officer had a lok at my visa and stamped the UAE entry stamp on my passport. Then I went through customs and came out at the Arrivals section of the airport, where I took a seat. It was around 22:00 hours local time.

The airline handed over my checked-in luggage - a heavy trolley bag - at the Dubai airport even though my connecting flight was with the same airline FlyDubai. Perhaps, it was because my layover was more than 24 hours long. The airline staff at Tirana only issued my boarding pass till Dubai. However, Abraham got his boarding passes till Kochi and he would get his luggage directly in Kochi.

I asked around but couldn’t find any cloak rooms to put my luggage. Soon Abraham joined me and I got to know he also has plans to roam around in the city.

At the airport, Abraham met a person whom he was talking in Malayalam. That person recommended that we withdraw at least 200 dirhams or AED (around 4,000 INR) in cash from ATM in order to roam around the city. Therefore, I withdrew 200 AED in cash.

We planned to spend the night at the airport and roam around in the city in the morning. We didn’t book a hotel and instead planned to sleep on the benches in the arrivals. I was hungry as I didn’t have lunch in my Tirana to Dubai flight, due to the lack of lacto-ovo vegetarian options on the flight. Therefore, I took a banana from the Costa Coffee shop located next to where we were sitting for 6 AED (equivalent to 120 rupees).

Unable to sleep, we came out of the airport in search of a place to eat. It was midnight and all the shops were closed. We were suggested by someone to check whether the McDonald’s inside the departures section was open. I was not sure whether we can just go inside the departures, as this is not allowed at Indian airports, unless you are catching a flight.

We met a lot of Indians and we asked where can we go for dining. One of them claimed that a restaurant near the Al-Qiyada metro station should be open. We walked 2 kilometers to reach that metro station. However, the restaurant wasn’t open, and so we had to return with disappointment. Abraham and I made some interesting conversations on the way to pass the time. I was starving at this point.

While returning to the Arrivals, we saw a restaurant open on the way. The name of the restaurant was ‘Food Castle Express’. It was located within 1 km of the Arrivals.

The restaurant had Indian food with reasonable prices. It was being run by people from Kerala and seemed to be a popular place among the airport staff. To give you an idea of prices, a plate of 3 Idlis was 5.5 AED (equivalent to 110 Indian Rupees), a cup of tea was 1 AED (20 Indian Rupees), a plate of Pav Bhaji was 8 AED (around 160 Indian Rupees), while Chicken Samosa, Aloo Samosa and Cheese Samosa, were 1.5 AED per piece, and Chicken Fried Rice was 12 AED.

I took the following:

Item	Price (AED)	Price (INR)
Idli (3 pcs)	5.5	110
1 Tea	1	20
Chhole Bhature	7.5	150
Total	14	280

We had this meal around 04:30 hours, and it kept me satiated literally for the whole day. Even though I was not in India, I was delighted to have Indian food. It is because we were having it after a long time because we were coming from Europe after a 3-week long stay, where every dish was bland.

After having our meal, we went back to the Arrivals section to catch some sleep. Abraham took a nap on the benches, while I failed to get any. In the meanwhile, I found out that the water inside the washroom was too hot to even wash my face.

Dubai Airport had a very good Wi-Fi which was a lifesaver. I didn’t have a local SIM card, so this helped me a lot in contacting home and passing time. The Wi-Fi has free unlimited access. You just need to simply select the “DXB Free Wifi” option, and it connects without asking for a phone number or OTP.

Abraham had slept for a couple of hours on the benches and woke up by 09:00. At this time, we exited the airport and were looking for a bus to drop us at some metro station. We found out that we need a card to use the public transport (such as buses, trams, metro) in Dubai. This card is called the Nol card. In order to obtain one, we had to go to a metro station. There must be other ways to obtain that card, but we didn’t know any.

The nearest metro station was Abu Hail Metro station - 2 km walk from the airport - which we had to cover by walking. It was not exactly fun as Dubai has a hot desert climate and the temperature was around 40°C, with sand coming into our eyes. To add to his, we had a lot of luggage. The bus stops were air-conditioned and sheltered, so we took shelter in one of the them while walking towards the metro station.

At the Abu Hail metro station, we took the silver Nol card (one for each), which was 25 AED (500 INR) with 19 AED balance. We planned to go to Burj Khalifa from here.

The metro from Abu Hail to Burj Khalifa was 5 AED. It was much cheaper than taking a taxi which would have added 25 AED as the basic fare. We reached the Burj Khalifa metro station in 25 minutes, following by walking to reach there. While clicking pictures, we realized that fitting the whole Burj Khalifa in a single frame is difficult. After all, it is the tallest building in the world!

After roaming around a bit, we went to the Dubai Mall, which was walking distance from the Burj Khalifa. The strategy was to spend as much time as possible inside such buildings to avoid the onslaught by the harsh weather. Abraham had a meal in the mall, while I didn’t eat anything due to lack of vegetarian options.

We still had 14 AED left in our Nol card at this point. Further, there is no facility to return the card and get the balance refunded. Therefore,we wanted to spend that amount before leaving the city. So, we planned a tram ride which will spend almost all the card balance, cover the city and does not require walking outside of air-conditioning. To take the tram, we went to Sobha Realty Metro Station and walked towards the tram station.

The tram ride was nice. If you ride the Dubai tram, be careful to check in and check-out using the Nol card at the tram station, otherwise you could be fined. We saw the Dubai’s Marina area from the tram. The tram takes 3 AED regardless of the station you check out from. We deboarded at the same station where we took the tram.

From there, we took a metro from DMCC station and returned to Abu Hail metro station. We went to Food Castle Express. I didn’t find a good vegetarian option there, so I skipped eating there and just took a chai.

After this, we returned to the airport to catch our respective flights. We didn’t face any language problems in Dubai. English was widely spoken. Most of my conversations were in Hindi, as I met many Indians and Pakistanis who knew Hindi. In fact, Abraham was talking to people in Malayalam (which is more popular in Dubai than Hindi). Overall, people were very nice which made us feel like home.

We had a nice time in Dubai, even though it was tiring due to lack of sleep and harsh weather.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/my-experience-attending-debconf22/ https://ravidwivedi.in/posts/my-experience-attending-debconf22/ Mon, 25 Jul 2022 20:22:10 GMT I just came out of what has been one of the most wonderful experiences of my life– DebConf in Kosovo. DebConf is the annual conference of Debian contributors from all around the world. This was my first time attending a DebConf ever, which presented a great opportunity for me to explore different cultures and food, along with making many new friends in the Debian community as it featured 210 attendees from 38 countries.

I registered and applied for bursary which was accepted by the DebConf bursary team. I applied for the Kosovo visa and after a long frustrating process of getting the visa, reached the venue of DebConf on 10th July 2022, which I have already written in detail. Getting Kosovo visa was already a great feat before DebConf and I am highly grateful to the organizers for putting their sincere efforts. Without this, it would not have been possible for us Indians to attend the event.

The Venue

The DebConf venue was Innovation and Training Park in Prizren, Kosovo. As soon as we reached the ITP campus, our eyes were treated with beautiful views of mountains and blue sky in all directions. The natural beauty of the place was stunning. You take a random photo of this place and it will come off as beautiful. This is how good it was. The fact that the campus was a German military base till December 2019 spiced up many people’s interests in the history of the place and there were some discussions regarding this.

The city of Prizren has a bridge over the Lumbardhi river which also features in the DebConf 22 logo. The bridge is a must visit place in Prizren, in my opinion. There was also a castle, which had views of the beautiful city of Prizren, but I missed visiting it.

The DebCamp held from 10 July to 16 July 2022 and the DebConf happened from 17th-24th July 2022. Most of the people came at the time of DebConf and the DebCamp days did not had so many attendees. By the time DebConf started, I already got adjusted to the campus.

Food

Being a non-meat eater, the first thing I was worried about before arriving at the venue was food as I read on the internet that Kosovo’s food is heavily based on meat. In my first lunch, I had only Veg Pasta multiple plates as most of the items in this lunch was meat-based. In the next two-three days, the availability of vegetarian food got better and I started enjoying the food. They started serving bean soup, vegetable soup, lentils and rice in the lunch time, expanding their vegetarian dishes. I enjoyed the food and it was way better than what I expected. The coconut and chocolate pastry served in the dinner time were delicious, and there was another delicious desert called Al-Baklava. Watermelon, Apples and Musk Melon, peach were frequently served fruits. The food got repetitive though after a point and it wasn’t as enjoyable, but I think it was still good as I am not sure if there is a lot of variety to expect from Kosovo in the vegetarian section.

The availability of drinking water was through the plastic bottles in the restaurant. In my opinion, this wastes a lot of plastic and is not an environment friendly option. Probably the locals were saying that the tap water is drinkable but it would still made sense to have a water filter in the dorms or water coolers etc.

DebConf

When the DebConf started on 17th July, new attendees started arriving in bigger numbers. Here I made many friends in the hacklab or when sitting over the lunch. As far as I remember, the most discussed topic was politics. They were very enthusiastic about telling their country’s history and the politics and asking me about the Indian culture, food etc.

The people were very very nice and I had a great time with the Debian community. The people were from very diverse backgrounds with each one having their unique stories on their involvement with Debian. On the other hand, there were people new to Debian and they asked my viewpoints on it, which gave me opportunities to propagate the Free Software Philosophy.

The participants ranged from being Google or Canonical employees to college students and high school students. I met a lot of students from Pristina, the capital of Kosovo, for whom the DebConf must have been a good opportunity. Since Debian is for everyone, there were many people from the non-technical side of Debian. I liked the fact that the attendees included people who didn’t know English, which gave more diversity.

I attended the talks whenever the topics seemed interesting to me. Since there were three tracks of talks happening simultaneously, I missed some of them. The dining, hacklab and the the accommodation were a bit far from each other. Further, it was mountaineous area and was not a easy breeze for me to get around in the campus so quickly.

There was a noisy hacklab and a silent hacklab. I used to hangout at the noisy hacklab as that gave me chance to meet new people. The noisy hacklab had free of charge coffee, while beer and water were given on the paid basis. Attendees could volunteer to be a bartender brewing and serving beer and coffee in the hacklab. The card games I played in the hacklab were very fun. Karl from the video team told me a long story on how much travelling video team has to do with the equipment and how hectic it is. I was totally ignorant about this aspect of DebConf before Karl told me this.

I didn’t spend a lot of time in my accommodation. I used to go there mainly for sleep and shower. The walls were thin, so there was an actual email sent in some debconf mailing list about not to talk loudly in the dorms. The rooms were spacious and had good air-conditioning. By the time DebConf started, dorms had a very good Wi-Fi connection.

I volunteered for some tasks for the video team. I was a talk meister in one talk– person who introduces the speaker and gives mic to people in audience who want to ask questions from the speaker, a camera operator in two of the talks.

Cheese and Wine Party

Cheese and Wine Party is a tradition of DebConf in which people bring food from their local areas to share with others. I brought Dal Samosa, which is a snack from my local place in India. This was, in general, liked by people in the party. I tried French alcoholic drink Pastis, which tasted like Fennel, and found it good. Cheese and Wine Party happened after the dinner, so I didn’t eat a lot. I had great conversations in the party with many people and it was fun.

DayTrip

The DayTrip was conducted by the DebConf to take time out of their schedule to visit some places in Kosovo. You can see all the options of DayTrips here.

I went to tour A: Bus tour of Eastern Kosovo. It covered Gadime Cave, Ulpiana Ancient City, Gracanica Monastery, Bear Sanctuary, Artana (Novoberda) Fortress. My feeling was that the day trip was trying to cover a lot of places, so it became more rushed. The places were beautiful and worth visiting in Kosovo. The tour featured an unexpected hiking in the Bear Sanctuary. The weather was sunny and very hot, so overall it was a tiring trip with beautiful places and good pictures.

The tour guide was excellent. He was informative and also took care of the people falling behind in the trip. We were joined by other guides, one at Marble Cave and the other at Gracanica Monestary. The guide at Marble cave was telling us about it in the Albanian langauge, which our tour guide was translating into English. The guides were very enthusiastic about the places they were telling us about. It was like an overdose of knowledge that day. We got a little late in our schedule and so we couldn’t really explore the Gracanica Monestary, we had to rush back to our destination after a few minutes.

Some photos from DayTrip taken by me:

Conference Dinner

The Conference Dinner was held outside of the DebConf venue in Alegria Hotel. Here the food was a disappointment for non-meat eaters/vegan as the restaurant only gave rice and some snacks. At the conference dinner, I liked the Flija. I still had some good company at the conference dinner so it was still fun.

Keysigning

In Debian, people sign each other’s keys in order to make a web of trust. It usually involves an ID check. People will usually ask you for your pasport to verify that it is really you. I saw many countries’ passport when keysigning, which was fun. After the ID check and verifying your key fingerprint, they sign your keys. Tobi helped me in configuring caff so that I can easily sign keys, rather than doing it manually.

Goodbye Kosovo

All in all, it was a great trip to Kosovo and a very unique one. I am already out of Kosovo as of writing this. I would like to visit Kosovo again in my life. It was so good. Thanks a lot to all the organizers, sponsors and people who made my time fun at the DebConf 22 in Kosovo.

Attending a DebConf is a learning opportunity on many fronts, be it the diversity among the people, cultures, food, opinion, or the software side of Debian. Writing about my experience has been a challenge in itself. Hopefully, I was able to pin down my experience as there was lot to share.

Looking forward to see you all in the next edition of DebConf which will take place in my country India.

Update on 06-August-2022: I got the reimbursement of all my travel expenses related to DebConf22 in just 4 days of filling the reimbursement form. So quick!

]]> Ravi Dwivedi https://ravidwivedi.in/posts/trip-to-prizren-bridge/ https://ravidwivedi.in/posts/trip-to-prizren-bridge/ Sat, 16 Jul 2022 00:00:00 GMT Yesterday I went to Prizren stone bridge, which is in Prizren, Kosovo. Prizren is often referred to as the cultural capital of Kosovo.

I am staying at the venue of DebConf 22 from where it is walkable distance, around 1.5 km.

I was going with my friends who were also attending the DebConf 22. We reached the bridge at around 20:30 hours time (Kosovo follows Central European Time). It is in the city center and the views were very beautiful. Lumbardhi river flows below the bridge with beautiful mountains surrounding on all sides.

The bridge also gives a nice view of Sinan Pasha Mosque , which is a very important monument of the Prizren city, built in the Ottoman architecture style.

Near the bridge, a person was roasting corn which costed €1 per piece. The corn was soft, but did not have many kernels.

Found this beautiful take home souvenier in the shape of Prizren stone bridge.

]]> Ravi Dwivedi https://mrkaran.dev/posts/dns-nomad/ https://mrkaran.dev/posts/dns-nomad/ Wed, 13 Jul 2022 18:30:00 GMT 5 seconds) while the writes of subsequent messages were instant. The following messages were instant for the next 30s, and then the write again took >5s. The request flow looks something like this: Go app in Nomad cluster (bridge mode) -> Kafka node in a Kafka cluster Both apps are inside the same VPC running on AWS EC2 instances. The Nomad task is running with network.mode=bridge, which means that there are some iptables rules configured to do SNAT/DNAT translation to forward packets from the default bridge network (nomad) to the default ethernet interface (ens5). We have a couple of other Nomad clusters in our environment and regularly use external EC2 instances to communicate, and we’ve never observed any slowdowns in our existing applications. This behaviour seemed something specific to Kafka. However, we discovered that we could not reproduce the high wait time issue in Kafka when the message was sent from a task running as host mode. So now, we had two conflicting things which made this issue strange: Issue happens in bridge mode, not in host mode. Issue happens only with Kafka nodes, not with any external services - even in bridge mode. We spent a lot of time dissecting our apps, turning on TRACE level logging for Kafka clusters and enabling debug mode in franz-go. One of the significant challenges for us was that even an idle Kafka cluster could be super chatty, producing ~1Mn records for a few minutes when run with TRACE. Cutting through the noise and finding the exact point where the slowdown happened turned out to be more difficult than expected. However, with the debug logs in franz-go, we’d arrived at a breakthrough which helped us narrow down the issue. We saw the write_wait time to be ~5s in the logs emitted by franz-go. The subsequent messages had write_wait as low as a few microseconds. What was puzzling was why Kafka took so much time to wait before writing the bytes to the underlying socket. We forked the franz-go library and added a bunch of our custom logs to figure out where and why exactly the slowdown happens. One issue in the logs emitted by franz-go was that no timestamps were attached to the logger. We added that and deployed the binaries with the patched version. This time we immediately found the logs, which pointed that it took ~5s from the time it initiated the connection to the Kafka broker node and the time it was able to connect to the node. The Fix# The node’s address was an internal hostname kafka-abc.private-zone.internal. We postulated that it could be a DNS resolver issue. We did a dig kafka-abc.private-zone.internal and instantly got the record. Maybe it’s cached? We decided to verify the /etc/resolv.conf till we waited for the TTL of the record to expire. Opening this seemingly innocent /etc/resolv.conf revealed what the issue was DNS indeed. nameserver bad nameserver good We had an unreachable nameserver address in the nameserver list. The first message has ~5s write timeout because /etc/resolv.conf has a 5s default timeout in case the nameserver is unreachable. Go’s DNS resolver picked up the second resolver, which cached the DNS records until the record’s TTL (30 seconds). Subsequent Kafka write messages on the topic worked without any issues in the TTL window. When the DNS record expires, rinse, and repeat. We later found the source of the bad nameserver came from our Nomad client initialising script. Takeaways# We use a custom Nomad client initialising script in our Nomad clusters to populate the chroot_env. Since, by default, the chroot provided by Nomad copies the/usr/ directory, we found that it increased the initial startup time of the alloc. It made sense to customise the chroot_env with the list of binaries and config files we would need. One of the configs happens to be /etc/resolv.conf, which DNS resolvers use to resolve queries. On the host, we have systemd-resolve running and /etc/resolv.conf is configured with the stub resolver address. However, since that address (127.0.0.53) is unreachable by the bridge network in Nomad, we mount our custom config, which looks like: nameserver 10.100.0.2 options edns0 trust-ad search ap-south-1.compute.internal The nameserver represents the AWS R53 resolver running in all VPCs if configured with enableDnsSupport in VPC settings. The above config ensures that all DNS queries for the tasks inside the bridge network go directly to the AWS R53 resolver. The resolver can do DNS Lookups for the private zones associated with the R53 in that VPC and forward all other hostnames to their own upstream DNS resolver. Fun Fact# If you don’t mount a custom /etc/resolv.conf, then DNS resolution is broken by default in any Nomad exec task. You can quickly reproduce it with this task definition: job "sleep" { datacenters = ["dc1"] type = "service" group "sleep" { count = 1 task "sleep" { driver = "exec" config { command = "bash" args = ["-c", "sleep infinity"] } } } } Run a Nomad agent in -dev mode: nomad agent -dev nomad run sleep.nomad When you exec inside the alloc: $ nomad alloc exec -i -t -task sleep 8b4a0a82 /bin/sh $ cat /etc/resolv.conf nameserver 127.0.0.1 $ dig mrkaran.dev ;; communications error to 127.0.0.1#53: connection refused I believe Nomad should bootstrap some DNS resolver or a relevant iptables rule for the exec tasks so that DNS can be resolved by default without the need to mount a custom config. For comparison, a docker also bootstraps the container with a customisable /etc/resolv.conf, and the settings can be specified either at runtime and fallback to the global settings in /etc/docker/daemon.json. I hope this post helps someone solve these weird DNS issues when running tasks with the bridge network and exec task driver in the Nomad cluster. Like they say: Fin!]]> At work, my colleagues and I stumbled upon a hair-pulling networking issue involving a specific problem when connecting to a Kafka cluster. We use the franz-go library in our Golang applications to interact with an external Kafka cluster. These Go apps are hosted inside a Nomad cluster and running with the exec driver.

The Issue#

Solving issues where a specific condition happens only sometimes is terribly difficult to debug because you need to reproduce the bug in a controlled environment. Our issue was that writing the first message to a Kafka topic took unusually high time (>5 seconds) while the writes of subsequent messages were instant. The following messages were instant for the next 30s, and then the write again took >5s.

The request flow looks something like this:

Go app in Nomad cluster (bridge mode) -> Kafka node in a Kafka cluster

Both apps are inside the same VPC running on AWS EC2 instances. The Nomad task is running with network.mode=bridge, which means that there are some iptables rules configured to do SNAT/DNAT translation to forward packets from the default bridge network (nomad) to the default ethernet interface (ens5).

We have a couple of other Nomad clusters in our environment and regularly use external EC2 instances to communicate, and we’ve never observed any slowdowns in our existing applications. This behaviour seemed something specific to Kafka. However, we discovered that we could not reproduce the high wait time issue in Kafka when the message was sent from a task running as host mode. So now, we had two conflicting things which made this issue strange:

• Issue happens in bridge mode, not in host mode.
• Issue happens only with Kafka nodes, not with any external services - even in bridge mode.

We spent a lot of time dissecting our apps, turning on TRACE level logging for Kafka clusters and enabling debug mode in franz-go. One of the significant challenges for us was that even an idle Kafka cluster could be super chatty, producing ~1Mn records for a few minutes when run with TRACE. Cutting through the noise and finding the exact point where the slowdown happened turned out to be more difficult than expected.

However, with the debug logs in franz-go, we’d arrived at a breakthrough which helped us narrow down the issue. We saw the write_wait time to be ~5s in the logs emitted by franz-go. The subsequent messages had write_wait as low as a few microseconds. What was puzzling was why Kafka took so much time to wait before writing the bytes to the underlying socket.

We forked the franz-go library and added a bunch of our custom logs to figure out where and why exactly the slowdown happens. One issue in the logs emitted by franz-go was that no timestamps were attached to the logger. We added that and deployed the binaries with the patched version. This time we immediately found the logs, which pointed that it took ~5s from the time it initiated the connection to the Kafka broker node and the time it was able to connect to the node.

The Fix#

The node’s address was an internal hostname kafka-abc.private-zone.internal. We postulated that it could be a DNS resolver issue. We did a dig kafka-abc.private-zone.internal and instantly got the record. Maybe it’s cached? We decided to verify the /etc/resolv.conf till we waited for the TTL of the record to expire. Opening this seemingly innocent /etc/resolv.conf revealed what the issue was DNS indeed.

nameserver bad
nameserver good

We had an unreachable nameserver address in the nameserver list. The first message has ~5s write timeout because /etc/resolv.conf has a 5s default timeout in case the nameserver is unreachable. Go’s DNS resolver picked up the second resolver, which cached the DNS records until the record’s TTL (30 seconds). Subsequent Kafka write messages on the topic worked without any issues in the TTL window. When the DNS record expires, rinse, and repeat. We later found the source of the bad nameserver came from our Nomad client initialising script.

Takeaways#

We use a custom Nomad client initialising script in our Nomad clusters to populate the chroot_env. Since, by default, the chroot provided by Nomad copies the/usr/ directory, we found that it increased the initial startup time of the alloc. It made sense to customise the chroot_env with the list of binaries and config files we would need.

One of the configs happens to be /etc/resolv.conf, which DNS resolvers use to resolve queries. On the host, we have systemd-resolve running and /etc/resolv.conf is configured with the stub resolver address. However, since that address (127.0.0.53) is unreachable by the bridge network in Nomad, we mount our custom config, which looks like:

nameserver 10.100.0.2
options edns0 trust-ad
search ap-south-1.compute.internal

The nameserver represents the AWS R53 resolver running in all VPCs if configured with enableDnsSupport in VPC settings.

The above config ensures that all DNS queries for the tasks inside the bridge network go directly to the AWS R53 resolver. The resolver can do DNS Lookups for the private zones associated with the R53 in that VPC and forward all other hostnames to their own upstream DNS resolver.

Fun Fact#

If you don’t mount a custom /etc/resolv.conf, then DNS resolution is broken by default in any Nomad exec task. You can quickly reproduce it with this task definition:

job "sleep" {
  datacenters = ["dc1"]
  type        = "service"
  group "sleep" {
    count = 1
    task "sleep" {
      driver = "exec"
      config {
        command = "bash"
        args    = ["-c", "sleep infinity"]
      }
    }
  }
}

Run a Nomad agent in -dev mode:

nomad agent -dev
nomad run sleep.nomad

When you exec inside the alloc:

$ nomad alloc exec -i -t -task sleep 8b4a0a82 /bin/sh
$ cat /etc/resolv.conf
nameserver 127.0.0.1
$ dig mrkaran.dev
;; communications error to 127.0.0.1#53: connection refused

I believe Nomad should bootstrap some DNS resolver or a relevant iptables rule for the exec tasks so that DNS can be resolved by default without the need to mount a custom config. For comparison, a docker also bootstraps the container with a customisable /etc/resolv.conf, and the settings can be specified either at runtime and fallback to the global settings in /etc/docker/daemon.json.

---

I hope this post helps someone solve these weird DNS issues when running tasks with the bridge network and exec task driver in the Nomad cluster.

Like they say:

Fin!

]]> Karan Sharma https://ravidwivedi.in/posts/pre-debconf-22/ https://ravidwivedi.in/posts/pre-debconf-22/ Wed, 13 Jul 2022 16:04:50 GMT DebConf is the annual conference of the Debian community. This year’s DebConf was held in Kosovo, a European country from the Balkans.

Debian sponsored me for the conference. Indian attendees needed a visa to enter Kosovo, which got complicated due to Kosovo not having any embassies in India (India doesn’t recognize Kosovo as an independent country). The organizing team came up with a solution - to send the required documents by email to Kosovo embassy in Tirana, Albania (neighboring country of Kosovo) and collecting the visas later by visiting the embassy in person. This was possible because Albania was granting visa-free entry for Indians during that time period. However, this was not the standard way to get a Kosovo visa and it was an exception made by the Ministry of Foreign Affairs of Kosovo for us.

The conference was to begin on the 10th of July. So, I booked my flight tickets from Delhi to Tirana for the 6th and bus tickets from Tirana to Prizren for the 10th of July. Further, I transferred the visa fee worth 40 Euros into the embassy’s bank account using wise.com. On the 9th of June, I emailed my documents to the Kosovo embassy in Tirana for my visa application. Here is the list of documents I sent:

• Filled and signed Visa Application form

• Scanned copy of my passport

• 1 photo of myself

• Invite letter and proof of travel, food and accommodation bursary.

• Confirmed return flight ticket from Delhi to Tirana

• Bus tickets from Tirana to Prizren

• Bank statement (last 3 months)

• Health insurance valid throughout the territory of the Republic of Kosovo

• Receipt of payment of 40 Euros for visa fee

I did not receive any acknowledgement of the receipt of my email from the embassy. On the 22nd of June (13 days after submitting the application), I wrote a follow-up email to the embassy asking for my visa application status. The embassy responded by saying:

Your name is not in the list of approved names to apply remotely. Therefore your case cannot be processed.

A couple of other Indian attendees reported getting the same response from the embassy. However, Utkarsh from the bursary team confirmed that our names were in the list. On the 27th of June, with 9 days to go before our flight to Albania, Praveen (a Debian Developer) sent an email to the Debian Project Leader Jonathan Carter, CCing all the prominent DebConf team members and Indian attendees, sharing our frustation with the process. He proposed Debian to reimburse tickets if we don’t get a response from the embassy.

On the other hand, Arianit Dobroshi from the organizing team was fairly confident about the approval and suggested us not to panic. In the same email, Praveen also proposed Debian to include visa fee in the bursary by default for the sponsored attendees, for which Jonathan shared his views in a public mailing list here.

As the travel date approached, I was panicking. I asked my travel agent to check on the cancellation fee for my flight. They told me it was around 15,000 INR. However, repeated assurances from Arianit made me stick to my plans. On July 3rd, he sent us a letter from Kosovo’s Ministry of Foreign Affairs stating that our visas were being processed in Tirana.

I wasn’t confident that the document was sufficient for boarding the flight. If I went to the airline staff at Delhi Airport and told them I was attending a conference in Kosovo, they would want to see whether I had a valid visa. Otherwise, they would refuse me boarding.

Therefore, I told the airline I was going to Albania. To my surprise, the airline did not ask me for a Kosovo visa upon seeing my conference invitation letter. Maybe they didn’t know about Kosovo, as India doesn’t recognize it?

After grilling me for some time, the airline gave me my boarding pass. The next stop was immigration. However, the immigration officer only asked me where I was going, to which I replied, “Albania,” and they asked me to show a visa. I told them that Albania was visa-free. So, they only checked whether Indians required a visa to visit Albania. I was elated after clearing immigration. This was followed by boarding the flight.

I had a connecting flight from Dubai. Here, I met other DebConf attendees coming from Kochi and Mumbai. Before boarding the flight, we were asked once again about our purpose for visiting Albania, to which we showed our invitation letters. Again, they didn’t notice that the conference was in Kosovo.

On the 7th of July, while we were in Tirana, we received the following email from one of the conference organizers, Arianit:

For Tirana applicants,

We got notification that visas have arrived in Tirana. Please show up at the Consulate tomorrow at 08:00 to get them stamped. They work until 13:30 tomorrow. We are trying to find a solution for people arriving in Tirana at 14:00 apparently.

Regards, Arianit

Next day, on the 8th of July, we went to the Kosovo embassy early in the morning and got our visas. On the 10th, we had a scenic morning bus trip from Tirana to Prizren.

I am at ITP Prizren—the venue of DebConf22—writing this post. It feels like a miracle to be here in Prizren attending DebConf. Looking forward to working on some projects and meeting nice people.

Lastly, I would like to thank the organizing team and the country of Kosovo.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/first-impressions-of-albania/ https://ravidwivedi.in/posts/first-impressions-of-albania/ Wed, 06 Jul 2022 00:00:00 GMT I landed in Tirana, Albania today, which is a country in the balkans region of the Europe.

My first impressions are:

• Tirana is neat and clean city with beautiful landscapes surrounding it.

• Locals are very patient and hospitable.

• Streets are not very crowded and footpaths are wide.

• The city has bars and cafes all around the place.

• It is a well planned city.

• It is cheaper than most of the places in Europe.

My first impression is that this is a very beautiful and hospitable place. Although, people who don’t eat meat can have some difficulties in finding good vegetarian food.

I hope to have fun in this Albania trip.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/is-gnu-linux-hard-for-beginners/ https://ravidwivedi.in/posts/is-gnu-linux-hard-for-beginners/ Tue, 28 Jun 2022 12:17:49 GMT GNU/Linux operating systems (if you heard about Linux operating systems, I am talking about the same. Read Why GNU/Linux if you are curious about the details.) have a reputation of being hard to use for non-techies. Some of the popular examples of GNU/Linux distributions are: Ubuntu, Debian, Fedora etc. I repeatedly hear that they are hard to install or knowledge of command line is necessary to use them and therefore only advanced users use GNU/Linux. This is not true. Depending on your requirements, you can choose a distro which does not require a lot of technical skills to use.

First of all, I would like to point out why I advertise GNU/Linux operating systems. I promote the idea of Free Software and when I suggest people to use freedom-respecting software, the obvious choice for a desktop operating system is a GNU/Linux operating system. There are many non-GNU Linux based operating systems as well and I am fine with people using them, but since I never used them, I suggest what I have used.

What I think is that GNU/Linux is for everybody. There is so much choice here that it is practically never ending. Whether you want to use as a normal user who just want a stable operating system or you are someone who likes to hack around and break things while learning something new, GNU/Linux is for you. Proprietary operating systems are full of malware, so it is good time to switch to a GNU/Linux distro, which are usually built by a community, everything is public and open for all to see, respect users’ freedom and privacy, plus they are usually far more secure than proprietary software, by design.

Whether it is school labs, a company’s production machine, personal computers, servers, Raspberry Pi, human right defenders, journalists, GNU/Linux is for everybody. The choice of the GNU/Linux distro depends on your use case and threat model. It is not like Windows that everyone is using the same thing with same appearance. Different people choose different distributions with very different appearances and customization, depending on their taste.

How to choose a GNU/Linux distribution for yourself

There is a huge number of GNU/Linux distros to choose from that it can be overwhelming to research all the choices. Librehunt website can ease your search by asking a few questions on what type of distro you want and then suggests you with a few distros matching your criteria. Popular GNU/Linux distributions like Ubuntu, Debian, Fedora, has a lot of user support by the community. If you use Ubuntu, for example, and you are stuck with some technical problem, chances are it is already answered on their Askubuntu forum. If not, then you can ask your question there. Many Free Software communities provide technical support. GNU/Linux distros usually have large communities behind them and work on the operating system of their choice in collaboration. Here are a few suggestions of mine for a user-friendly distro to start with: Debian, Ubuntu, Zorin OS, GNU/Linux Mint, Fedora. I haven’t tried all of them but I have heard good feedback about Zorin OS, Linux Mint and Fedora.

After you have selected a distro, the next step is to choose a Desktop Environment. These differ in their appearances and philosophy. For example, if you are planning to run GNU/Linux on old hardware, then XFCE is a suitable Desktop Environment, which is lightweight and does not consume a lot of resources. GNOME and KDE Plasma are very popular Desktop Environments with large communities behind them.

Installation

The installation of a distro depends on your hardware and the distro you choose. The distro you are installing might have installation guide on its website. For many distros, the installation is kind of the same. Arch Linux is for a bit advanced users where the installation differs from, say Ubuntu.

You will find tons of blogs and video tutorials to install distro of your choice.

Many laptops ship with GNU/Linux preinstalled. Some examples are: Indian vendor Mostly Harmless, Purism devices, Ministry of Freedom project in the UK, Lenovo selling laptops with Ubuntu pre-installed and Fedora pre-installed, Clevo selling laptops with Debian pre-installed, Manjaro has also partnered with some hardware manufacturers to ship with Manjaro out of the box, System 76 laptops, etc.

I suggest you to do some research and choose a computer with GNU/Linux preinstalled for the next time.

Learning about the GNU/Linux operating system

Many blogs and Youtube channels exist where you can learn about the distros. If you are curious to learn the command line, you will find tons of resources on the internet for learning. There are many books written on the command line which you can choose for your learning. Whatever distro you are using, Arch Wiki can be a handy guide.

I started with Debian as a beginner and I learnt some command line just for fun and curiosity, but it is not mandatory for a Debian user. I found the operating system easy and intuitive to use. I have used Manjaro for a couple of months and PureOS for around half a year. Whenever I am stuck, I usually get help from the internet.

Conclusion

All in all, GNU/Linux distros are highly customizable and give users choice and freedom to use what they want. There are user-friendly distros like Ubuntu and Debian with good user support and large communities behind them so that you can always ask for help. I hope this will post will strike down any fear or doubts you might have in starting using GNU/Linux.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/what-is-debian/ https://ravidwivedi.in/posts/what-is-debian/ Sun, 26 Jun 2022 08:25:52 GMT Debian as an operating system

Debian GNU/Linux is a Free Software (software which respects user’s freedom to run, study, modify, share and share the modified versions) operating system. Debian is committed to Free Software philosophy, as software freedom is deeply embedded in its social contract. Debian’s official image ships with only free software including firmware, and the official repositories also contain only Free Software. If users need nonfree drivers, then separate images with nonfree firmware are also available on the Debian website.

Debian operating system has three branches: stable, testing and unstable. Debian explains that the main goal of the project is to develop the stable branch. Other branches are only a means for that end. Of course, many users might still find it suitable to install testing or unstable, keeping in mind that these might break.

Debian is known for its stability and its universality. The stability of Debian stable branch comes from the fact that it is extensively tested before the release as the testing branch. The three branch system allows the user to choose the desired stability. The stable branch gets only security updates and mission critical bug fixes. So, the packages in stable branch might not be the latest version, but they are tested thoroughly before putting there. Debian Stable users can download recent releases of some software from the Debian backports repository. Stable branch is a recommended one for a company’s production machine or school labs and personal computers too.

Debian can run on all types of hardware and supports 9 CPU architectures and some others unofficially as well. It can run on your personal computer, a school computer, old hardware, servers, Raspberry Pi and similar devices, IoT devices, etc. Whatever piece of hardware you have, you can put Debian on it, and that is why Debian’s tagline is ‘The Universal Operating System’. This is far more than is available for any other GNU/Linux distribution.

Many popular GNU/linux distributions are based on Debian. Examples are: Ubuntu, PureOS, Tails OS etc.

People behind Debian

Debian’s biggest feature is the community behind it and how that community makes decisions. The guiding principles of Debian community are the Debian Social Contract and the Debian Free Software guidelines.

In short, the Debian Social contract promises the following to its users: 1) 100% Free Software; 2) Debian project will give back to Free Software community; 3) Problems won’t be hidden; 4) Users are priority; 5) providing support for nonfree packages to users, if required.

The track record of 29 years of the project says that these promises have never been broken. It is an independent project made by volunteers all around the globe. It has a democratic decision-making structure as directed by its constitution.

Companies or individuals sponsoring Debian project do not get any say in the decisions of the Debian project just because they gave money to the project.

Debian project’s democratic decision making along with software freedom of the operating system makes it one of the most secure operating systems, since Debian does not have a single point of failure.

This works as follows:

Every Debian Developer’s gpg keys are signed by at least two other existing Debian developers. It is a decentralized trust model and forms a web of trust. Every package uploaded to Debian is compiled from its source code and signed by the private gpg keys of a Debian Developer. Only the owner of the keys can have their private keys. A maintainer of a Debian package abusing their position to upload malicious code intentionally will harm their own reputation in the society as every upload is public and it will stay there for lifetime. Compare this to a centralized corporation where they can upload malicious code in the packages but this commit is not public.

How can we make sure that the binaries match the source code? Debian is also working on reproducible builds, so that users can build the package on their own and verify that the binary actually matches the source code. Purism explained the concept very well in their blog post.

Debian has a separate LTS team which provides support to Debian stable releases for at least 5 years. This can be helpful for companies and organizations who want support in their Debian usage.

In conclusion, Debian is a freedom-respecting operating system maintained and developed by its users around the globe. It is not controlled by a single entity and this makes it hard to compromise, making it one of the most secure operating systems on the earth.

My experience

I started using Debian in August 2021 and I have tried all the branches of Debian: stable, testing and unstable. I was not using only Debian in this time frame and have switched to PureOS for around 6 months. Personally, I faced no problems in using Debian. I found that even testing branch is stable enough to be my daily driver. Currently, I am running the stable branch of Debian Bullseye, along with adding backports to my sources file. The Desktop environment that has suited me the most as of now is KDE Plasma. I love it and whenever I switch to any other Desktop Environment, I do not feel at home.

Read further

Want to install Debian in your computer? or curious about how to contribute to Debian? Please check:

• Get Debian for your device.

• Contribute to Debian.

• An Indian shop sells hardware that can run fully free version of Debian.

• Buy a computer with Debian pre-installed.

• Clevo is shipping laptops with Debian out-of-the-box.

• Librem hardware(like laptops and mobile phones) can run Debian.

• If you have any technical questions, please feel free to ask any of the Debian user support channels or Free Software Community of India.

]]> Ravi Dwivedi https://shrirangkahale.com/posts/albony-mirror/ https://shrirangkahale.com/posts/albony-mirror/ Fri, 24 Jun 2022 16:35:04 GMT Shrirang Kahale https://shrirangkahale.com/posts/apt-ipv6/ https://shrirangkahale.com/posts/apt-ipv6/ Mon, 16 May 2022 09:42:24 GMT Shrirang Kahale https://mrkaran.dev/posts/nomad-networking-explained/ https://mrkaran.dev/posts/nomad-networking-explained/ Fri, 13 May 2022 18:40:55 GMT 6379/tcp under PORTS. To verify, I can use redis-cli to connect to this host: $ redis-cli -p 49153 127.0.0.1:49153> ping PONG Fantastic! Now, let’s stop this container and see how we can do the same in Nomad. job "redis" { datacenters = ["dc1"] type = "service" group "redis" { count = 1 network { mode = "host" port "redis" { to = 6379 } } task "redis" { driver = "docker" config { image = "redis:7" ports = ["redis"] } resources { cores = 1 memory = 256 } } } } This is a barebones definition of how to run Redis on Nomad. We’re not dealing with any volume mounts, custom config etc here - the idea is to only learn networking concepts in Nomad! Save the above file as job.nomad and deploy to the cluster with: nomad run job.nomad Within a few lines of config, we have a Docker container running, which exposes a dynamic port 23522. We can connect to it via redis-cli on our host: $ redis-cli -p 23522 127.0.0.1:23522> ping PONG NOTE: It’s important to have ports in your task.config section. Nomad passes this information to the docker daemon running on the host. So unless you specify which ports to advertise in the container, it won’t know whether to expose 6379 or not. This can be easily verified with docker inspect: # without `ports` $ docker inspect fc32a4ffd148 -f "{{json .NetworkSettings.Ports }}" | jq '.' { "6379/tcp": null } # with `ports` $ docker inspect 0421101d0228 -f "{{json .NetworkSettings.Ports }}" | jq '.' { "6379/tcp": [ { "HostIp": "127.0.0.1", "HostPort": "31930" } ], "6379/udp": [ { "HostIp": "127.0.0.1", "HostPort": "31930" } ] } Exposing Static ports# A less common scenario is to bind an application to a static port on the host. We’ll cover an example of when you want to do that a bit later, but this is generally not widely used because in a cluster your application can “float” around and the idea of sticking to a port is not useful. However, there’s a way for us to do that by simply adding a static line in our port block: network { port "redis" { to = 6379 static = 6379 } } When we deploy the same file again, we can see the port allocation has changed from dynamic to the static port we assigned. It’s your job to ensure no other applications are listening on that same interface and port because that’s bound to cause conflicts. Scenario 2: Communicate to Redis within the same group# For this scenario, we are going to assume there’s a Go application that needs to talk to Redis. However, in this scenario, the Redis is sort of like an ephemeral cache, so it’s okay to deploy both of them in the same Task Group. A Group can contain multiple tasks. What's important to know here is that a group will always have its own shared network namespace. This means, that if you have 2 tasks in the group, they both will have access to the same network namespace. This allows both tasks to talk to each other on the same network interface. job "hello" { datacenters = ["dc1"] type = "service" group "app" { count = 1 network { mode = "host" port "app" { to = 8080 static = 8080 } port "redis" { to = 6379 static = 6379 } } task "redis" { driver = "docker" config { network_mode = "host" image = "redis:7" ports = ["redis"] } resources { cores = 1 memory = 256 } } task "app" { driver = "docker" env { DEMO_REDIS_ADDR = "${NOMAD_ADDR_redis}" } config { network_mode = "host" image = "mrkaran/hello-app:1.0.0" ports = ["app"] } resources { cores = 1 memory = 512 } } } } Key Points: You can see we have defined task app and task redis under the same group, app. This means that Nomad will co-locate both of these tasks on the same client (because they tend to share not just the same network namespace but a common allocation directory as well - which makes it super easy to share files across tasks). We are using NOMAD_ADDR_redis to get the IP:Port combination for the redis task. This gets injected at runtime by Nomad. You can find a list of runtime variables here. This is ideal for quick tests/dev setup where you don’t want the overhang of Service Discovery etc and want to connect to your applications in the least friction possible. The above config is suitable if you’re migrating from docker-compose based environments, you can use this template for your services. The biggest limitation of this approach is that it’s using a host network so it’s not possible to set up any kind of Access Controls on it. This effectively means that nothing prevents any other application on the cluster to talk to these ports. Scenario 3: Communicate across different groups# Task Groups are useful if you have related tasks (like the init task where you wanna fetch the files before the task starts). But a drawback of using group is that you can’t scale the tasks independently. In the above example, we placed Redis and App in the same group, but that means if you increase count of same group to scale the app, you end up scaling Redis containers too. This is undesirable as Redis may not to scale proportionally to app. The way to do create multiple groups is to split the tasks into their own individual groups: job "hello" { datacenters = ["dc1"] type = "service" group "app" { count = 1 network { mode = "host" port "app" { to = 8080 static = 8080 } } task "app" { driver = "docker" env { DEMO_REDIS_ADDR = "localhost:6379" } config { image = "mrkaran/hello-app:1.0.0" ports = ["app"] } resources { cores = 1 memory = 512 } } } group "redis" { count = 1 network { mode = "host" port "redis" { to = 6379 static = 6379 } } task "redis" { driver = "docker" config { image = "redis:7" ports = ["redis"] } resources { cores = 1 memory = 256 } } } } When you submit this job, you get 2 allocation IDs (each group creates one alloc). The key point here is that both of these groups have their own network namespace. So, we don’t really have any way to reach the other application (we can’t really rely on the host network, because there’s no guarantee that both of these groups will be deployed on the same node). In the previous example, we saw how Nomad exposes runtime variables that contained information about all tasks in other groups. But now since the groups are separate, the app container has no idea about redis (or vice-versa): env | grep NOMAD NOMAD_REGION=global NOMAD_CPU_LIMIT=4700 NOMAD_IP_app=127.0.0.1 NOMAD_JOB_ID=hello NOMAD_TASK_NAME=app NOMAD_SECRETS_DIR=/secrets NOMAD_CPU_CORES=1 NOMAD_NAMESPACE=default NOMAD_ALLOC_INDEX=0 NOMAD_ALLOC_DIR=/alloc NOMAD_JOB_NAME=hello NOMAD_HOST_IP_app=127.0.0.1 NOMAD_SHORT_ALLOC_ID=a9da72dc NOMAD_DC=dc1 NOMAD_ALLOC_NAME=hello.app[0] NOMAD_PORT_app=8080 NOMAD_GROUP_NAME=app NOMAD_PARENT_CGROUP=nomad.slice NOMAD_TASK_DIR=/local NOMAD_HOST_PORT_app=8080 NOMAD_MEMORY_LIMIT=512 NOMAD_ADDR_app=127.0.0.1:8080 NOMAD_ALLOC_PORT_app=8080 NOMAD_ALLOC_ID=a9da72dc-94fc-6315-bb37-63cbeef153b9 NOMAD_HOST_ADDR_app=127.0.0.1:8080 Service Discovery# This is where things get interesting. The app group needs to discover redis before connecting to it. There are multiple ways to do that, but we’ll cover 2 standard ways which are more common. Using Nomad native service discovery# This is a feature launched in Nomad 1.3. Up until this release, Nomad had to rely on Consul for this. But with native service discovery built in Nomad, things are much simpler. Let’s make the following changes to our job file. In each group, we’ll add a service definition: group "app" { count = 1 network { mode = "host" port "app" { to = 8080 } } service { name = "app" provider = "nomad" port = "app" } // task is the same } group "redis" { count = 1 network { mode = "host" port "redis" { to = 6379 } } service { name = "redis" provider = "nomad" port = "redis" } // task is the same } So, we added a new service block and got rid of static ports. Well, there’s no need to bind to static ports when we’re using service discovery. After submitting the job, we can use the nomad service list command to ensure the services are registered with Nomad. nomad service list Service Name Tags app [] redis [] To find out details about a particular service, we can use nomad service info: $ nomad service info app Job ID Address Tags Node ID Alloc ID hello 127.0.0.1:29948 [] d92224a5 5f2ac51f $ nomad service info redis Job ID Address Tags Node ID Alloc ID hello 127.0.0.1:22300 [] d92224a5 8078c9a6 Perfect! We can see the dynamic port assignment in each of the services. To use this config in our app, we will template it: task "app" { driver = "docker" template { data = < Nomad is a general-purpose cluster orchestrator and scheduler. Up until Nomad 1.3 was released, it had no native support for discovering other applications running in the cluster. This is sort of a very elementary requirement when scheduling tasks in a cluster. Nomad relies on Consul to discover other “services” and has first class support for registering and fetching service records which makes things easier. Consul provides the records via various mechanisms such as a REST API, DNS and Consul Templates which render the exact IP/Port of the service in a Go template that can be injected into your application.

I’ve been using Nomad since quite some time (both at work and for my self hosted instance) however I’ve often tripped when it comes to Networking. Nomad has a lot of simple concepts and it all “clicks” once you understand and recognise various patterns that can be used to connect the applications. A major learning curve for someone new to Nomad and trying to integrate Consul is that the person now has to first understand how Consul works, deploy a Consul cluster and this creates a lot of friction amongst newcomers to Nomad. Nomad 1.3 solves a part of this issue (i.e. no need to run Consul for basic service discovery) and is a great fit for just getting started with Nomad based networking. However, in this post I’d like to go through all the different networking patterns I’ve known or used in production and make an attempt at simplifying these concepts for Nomad beginners.

I'll be running a single node Nomad on my dev machine. The instructions to do that can be seen here.

Scenario 1: Expose an application on the host#

We’ll start off with the simplest usecase: You have a redis container and you want to expose that to the host. The docker run equivalent to what we wanna do is:

docker run --rm -p=6379 redis:7

This command exposes a dynamic port on your host. To see what exactly is the port number, you can do docker ps and find out an output similar to 0.0.0.0:49153->6379/tcp under PORTS.

To verify, I can use redis-cli to connect to this host:

$ redis-cli -p 49153                
127.0.0.1:49153> ping
PONG

Fantastic! Now, let’s stop this container and see how we can do the same in Nomad.

job "redis" {
  datacenters = ["dc1"]
  type        = "service"

  group "redis" {
    count = 1

    network {
      mode = "host"
      port "redis" {
        to = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:7"
        ports = ["redis"]
      }

      resources {
        cores  = 1
        memory = 256
      }
    }
  }
}

This is a barebones definition of how to run Redis on Nomad. We’re not dealing with any volume mounts, custom config etc here - the idea is to only learn networking concepts in Nomad!

Save the above file as job.nomad and deploy to the cluster with:

nomad run job.nomad

Within a few lines of config, we have a Docker container running, which exposes a dynamic port 23522.

We can connect to it via redis-cli on our host:

$ redis-cli -p 23522
127.0.0.1:23522> ping
PONG

NOTE: It’s important to have ports in your task.config section. Nomad passes this information to the docker daemon running on the host. So unless you specify which ports to advertise in the container, it won’t know whether to expose 6379 or not.

This can be easily verified with docker inspect:

# without `ports`
$ docker inspect fc32a4ffd148 -f "{{json .NetworkSettings.Ports }}" | jq '.'
{
  "6379/tcp": null
}

# with `ports`
$ docker inspect 0421101d0228 -f "{{json .NetworkSettings.Ports }}" | jq '.'
{
  "6379/tcp": [
    {
      "HostIp": "127.0.0.1",
      "HostPort": "31930"
    }
  ],
  "6379/udp": [
    {
      "HostIp": "127.0.0.1",
      "HostPort": "31930"
    }
  ]
}

Exposing Static ports#

A less common scenario is to bind an application to a static port on the host. We’ll cover an example of when you want to do that a bit later, but this is generally not widely used because in a cluster your application can “float” around and the idea of sticking to a port is not useful. However, there’s a way for us to do that by simply adding a static line in our port block:

    network {
      port "redis" {
        to     = 6379
        static = 6379
      }
    }

When we deploy the same file again, we can see the port allocation has changed from dynamic to the static port we assigned. It’s your job to ensure no other applications are listening on that same interface and port because that’s bound to cause conflicts.

Scenario 2: Communicate to Redis within the same group#

For this scenario, we are going to assume there’s a Go application that needs to talk to Redis. However, in this scenario, the Redis is sort of like an ephemeral cache, so it’s okay to deploy both of them in the same Task Group.

If you don't know the difference between a Task and Group, here's a very primitive explanation but please read the docs for more clarity.

A Group can contain multiple tasks. What's important to know here is that a group will always have its own shared network namespace. This means, that if you have 2 tasks in the group, they both will have access to the same network namespace. This allows both tasks to talk to each other on the same network interface.

job "hello" {
  datacenters = ["dc1"]
  type        = "service"

  group "app" {
    count = 1

    network {
      mode = "host"
      port "app" {
        to     = 8080
        static = 8080
      }
      port "redis" {
        to     = 6379
        static = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        network_mode = "host"
        image        = "redis:7"
        ports        = ["redis"]
      }

      resources {
        cores  = 1
        memory = 256
      }
    }


    task "app" {
      driver = "docker"
      env {
        DEMO_REDIS_ADDR = "${NOMAD_ADDR_redis}"
      }

      config {
        network_mode = "host"
        image        = "mrkaran/hello-app:1.0.0"
        ports        = ["app"]
      }

      resources {
        cores  = 1
        memory = 512
      }
    }
  }
}

Key Points:

• You can see we have defined task app and task redis under the same group, app. This means that Nomad will co-locate both of these tasks on the same client (because they tend to share not just the same network namespace but a common allocation directory as well - which makes it super easy to share files across tasks).
• We are using NOMAD_ADDR_redis to get the IP:Port combination for the redis task. This gets injected at runtime by Nomad. You can find a list of runtime variables here.
• This is ideal for quick tests/dev setup where you don’t want the overhang of Service Discovery etc and want to connect to your applications in the least friction possible.

The above config is suitable if you’re migrating from docker-compose based environments, you can use this template for your services. The biggest limitation of this approach is that it’s using a host network so it’s not possible to set up any kind of Access Controls on it. This effectively means that nothing prevents any other application on the cluster to talk to these ports.

Scenario 3: Communicate across different groups#

Task Groups are useful if you have related tasks (like the init task where you wanna fetch the files before the task starts). But a drawback of using group is that you can’t scale the tasks independently. In the above example, we placed Redis and App in the same group, but that means if you increase count of same group to scale the app, you end up scaling Redis containers too. This is undesirable as Redis may not to scale proportionally to app.

The way to do create multiple groups is to split the tasks into their own individual groups:

job "hello" {
  datacenters = ["dc1"]
  type        = "service"

  group "app" {
    count = 1

    network {
      mode = "host"
      port "app" {
        to     = 8080
        static = 8080
      }
    }

    task "app" {
      driver = "docker"
      env {
        DEMO_REDIS_ADDR = "localhost:6379"
      }

      config {
        image = "mrkaran/hello-app:1.0.0"
        ports = ["app"]
      }

      resources {
        cores  = 1
        memory = 512
      }
    }
  }

  group "redis" {
    count = 1

    network {
      mode = "host"
      port "redis" {
        to     = 6379
        static = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:7"
        ports = ["redis"]
      }

      resources {
        cores  = 1
        memory = 256
      }
    }
  }
}

When you submit this job, you get 2 allocation IDs (each group creates one alloc). The key point here is that both of these groups have their own network namespace. So, we don’t really have any way to reach the other application (we can’t really rely on the host network, because there’s no guarantee that both of these groups will be deployed on the same node).

In the previous example, we saw how Nomad exposes runtime variables that contained information about all tasks in other groups. But now since the groups are separate, the app container has no idea about redis (or vice-versa):

env | grep NOMAD
NOMAD_REGION=global
NOMAD_CPU_LIMIT=4700
NOMAD_IP_app=127.0.0.1
NOMAD_JOB_ID=hello
NOMAD_TASK_NAME=app
NOMAD_SECRETS_DIR=/secrets
NOMAD_CPU_CORES=1
NOMAD_NAMESPACE=default
NOMAD_ALLOC_INDEX=0
NOMAD_ALLOC_DIR=/alloc
NOMAD_JOB_NAME=hello
NOMAD_HOST_IP_app=127.0.0.1
NOMAD_SHORT_ALLOC_ID=a9da72dc
NOMAD_DC=dc1
NOMAD_ALLOC_NAME=hello.app[0]
NOMAD_PORT_app=8080
NOMAD_GROUP_NAME=app
NOMAD_PARENT_CGROUP=nomad.slice
NOMAD_TASK_DIR=/local
NOMAD_HOST_PORT_app=8080
NOMAD_MEMORY_LIMIT=512
NOMAD_ADDR_app=127.0.0.1:8080
NOMAD_ALLOC_PORT_app=8080
NOMAD_ALLOC_ID=a9da72dc-94fc-6315-bb37-63cbeef153b9
NOMAD_HOST_ADDR_app=127.0.0.1:8080

Service Discovery#

This is where things get interesting. The app group needs to discover redis before connecting to it. There are multiple ways to do that, but we’ll cover 2 standard ways which are more common.

Using Nomad native service discovery#

This is a feature launched in Nomad 1.3. Up until this release, Nomad had to rely on Consul for this. But with native service discovery built in Nomad, things are much simpler. Let’s make the following changes to our job file. In each group, we’ll add a service definition:

  group "app" {
    count = 1

    network {
      mode = "host"
      port "app" {
        to = 8080
      }
    }

    service {
      name     = "app"
      provider = "nomad"
      port     = "app"
    }
    // task is the same
  }

  group "redis" {
    count = 1

    network {
      mode = "host"
      port "redis" {
        to = 6379
      }
    }

    service {
      name     = "redis"
      provider = "nomad"
      port     = "redis"
    }
    // task is the same
  }

So, we added a new service block and got rid of static ports. Well, there’s no need to bind to static ports when we’re using service discovery.

After submitting the job, we can use the nomad service list command to ensure the services are registered with Nomad.

nomad service list    
Service Name  Tags
app           []
redis         []

To find out details about a particular service, we can use nomad service info:

$ nomad service info app      
Job ID  Address          Tags  Node ID   Alloc ID
hello   127.0.0.1:29948  []    d92224a5  5f2ac51f
$ nomad service info redis
Job ID  Address          Tags  Node ID   Alloc ID
hello   127.0.0.1:22300  []    d92224a5  8078c9a6

Perfect! We can see the dynamic port assignment in each of the services. To use this config in our app, we will template it:

    task "app" {
      driver = "docker"

      template {
        data = <<EOH
{{ range nomadService "redis" }}
DEMO_REDIS_ADDR={{ .Address }}:{{ .Port }}
{{ end }}
EOH

        destination = "secrets/config.env"
        env         = true
      }

      config {
        image = "mrkaran/hello-app:1.0.0"
        ports = ["app"]
      }

      resources {
        cores  = 1
        memory = 512
      }
    }

We added the template stanza which will interpolate the env variables in the container. We loop over nomadService and get the address and port of the redis service. This makes it convenient for tasks on other nodes to discover each other.

Using Consul Service Discovery#

Just by tweaking provider in our service block, we can use the Consul agent for service discovery.

    service {
      name     = "app"
      provider = "consul"
      port     = "app"
    }


    task "app" {
      driver = "docker"

      template {
        data = <<EOH
{{ range service "redis" }}
DEMO_REDIS_ADDR={{ .Address }}:{{ .Port }}
{{ end }}
EOH

Ensure that you're running Consul and have connected Nomad to it. Please refer to docs for the same.

Since now we are using consul for registering services, we have to loop over service instead of nomadService. The rest of the things remain pretty much the same. I really like how with just 2 lines of code you can switch between Nomad/Consul for discovering services.

Now, of course, there are certain advantages to using Consul:

• You can query the address of the service with DNS:

doggo redis.service.consul @tcp://127.0.0.1:8600
NAME                    TYPE    CLASS   TTL ADDRESS     NAMESERVER     
redis.service.consul.   A       IN      0s  172.20.10.3 127.0.0.1:8600  

• Define health checks. Since it’s a new feature, health checks on Nomad service aren’t there but there’s a GitHub issue open for the same.

Update (2024): Native health checks are now available in Nomad since version 1.4 (released October 2022). This means you can define health checks directly in your Nomad service definitions without requiring Consul. You can read more about this in the Nomad documentation.

• Accessible by applications outside Nomad. In case consul is used by other applications outside of the Nomad cluster, they can still get their address (using DNS or REST APIs)

However, Nomad native service discovery is perfect for local setups and even smaller use-cases in production because it eliminates the need of running Consul in your stack which is a big thing!

Scenario 4: Restricting access to certain namespaces#

In all the above scenarios, we found that the service gets exposed to the local Nomad client. In case you’re running multiple namespaces on your cluster, you’d like to not expose them at all. In addition, you may want to express fine-grained controls on which application can access a particular service. All of this is possible via a Service Mesh. Nomad provides a way to have a “service mesh” via Consul Connect. Consul Connect can do mTLS and service authorization. Under the hood, it’s an Envoy proxy that runs alongside your app (sidecar is a fancy way to say that). The consul agent configures an Envoy configuration for you so it’s all pretty seamless.

For this to work, the first thing we need is a bridge network mode. This network model is actually a CNI plugin and needs to be installed separately in /opt/cni/bin. Follow the steps mentioned here.

    network {
      mode = "bridge"
      port "redis" {
        to = 6379
      }
    }

The service in redis is called as a Consul Connect Ingress:

    service {
      name     = "redis"
      provider = "consul"
      port     = "6379"
      connect {
        sidecar_service {}
      }
    }

It’s an empty block because we don’t need to define any upstream here. The rest of the values will be default values.

Next, we create a service for our app and that is a Consul Connect Egress:

    service {
      name     = "app"
      provider = "consul"
      port     = "app"
      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "redis"
              local_bind_port  = 6379
            }
          }
        }
      }
    }

Here we define an upstream for redis. If you notice closely, we are using a port number in Consul Connect Ingress. For some reason, if you use a named port instead of 6379 it doesn’t work. I am not entirely sure if it’s a bug or it’s intended to work like this.

So here, when the app wants to talk to redis, it talks to localhost:6379 which is the local port that the Envoy sidecar is listening to. We can verify that using netstat:

$ netstat -tulpvn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.2:19001         0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:23237           0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::8080                 :::*                    LISTEN      1/./hello.bin

The traffic is sent from this port to the other Envoy proxy on a port that it advertises (and Consul automagically configured). That Envoy proxy further sends the traffic to the redis container on port 6379. The proxied traffic is securley encrypted via mTLS and authorized (via Consul Intentions - not covered in this post).

Scenario 5: Exposing services to end-user#

In the first scenario, we discussed using static ports. Well, it turns out it’s super helpful if you want to define a Traffic Ingress service. Unlike K8s, Nomad doesn’t have any Ingress Controllers, so the best way is to deploy these web proxies on each node as a system job (which means it’s ensured to run on every client node) and bind them to a static port (say 443/80). Then, configure your Load Balancers and register all the Nomad nodes as target IPs and their ports would be the static port you define. These Ingress proxies (like HAProxy/Nginx) can then be used to talk to your application via any of the patterns we’ve mentioned above.

Typically, you’d want to use a “Host-Based” routing pattern for your ingress proxy to make a routing decision.

For eg, in case you have an a.example.org DNS record pointing to an ALB. Now, when the request comes to the ALB, it forwards to any one of the NGINX/HAProxy. For HAProxy to correctly route the traffic to `a service, you can use a “Host” header.

Summary#

These were some of the common networking patterns that I’m aware of. Since some of these concepts are not really straightforward I hope the explanation helped in bringing some clarity.

There’s much more to this topic like Consul Gateways and multiple kind of CNIs which tweak how networking happens in the cluster but those are some really advanced topics that are out of the scope for this post.

Fin!

]]> Karan Sharma https://www.evalapply.org/posts/which-clojure-codebases-to-read-how-and-why/index.html https://www.evalapply.org/posts/which-clojure-codebases-to-read-how-and-why/index.html Fri, 29 Apr 2022 00:00:00 GMT Aditya Athalye clojure howto whyto https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-2-functions-as-unix-tools/index.html https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-2-functions-as-unix-tools/index.html Wed, 27 Apr 2022 00:00:00 GMT Aditya Athalye bash unix functional_programming architecture https://www.evalapply.org/posts/why-and-how-i-use-org-mode/index.html https://www.evalapply.org/posts/why-and-how-i-use-org-mode/index.html Tue, 19 Apr 2022 00:00:00 GMT Aditya Athalye howto whyto writing emacs org_mode local_first notebooks knowledge_management tools_for_thought https://aboobacker.in/2022/04/12/postgres-transaction-pitfalls-for-rails-developers.html https://aboobacker.in/2022/04/12/postgres-transaction-pitfalls-for-rails-developers.html Tue, 12 Apr 2022 07:02:00 GMT Rails abstract lot of database stuff away using Active record which is very convenient. But the convenience can bite back if we are not careful enough.

Here I am going to list some common mistakes rails developers make and how to avoid them.

1. Having network calls on rails after_create/before_create callbacks

Active record provides callback methods to perform some operations based on the changes in data. it includes before_create, after_create, after_commit etc. Rails wrap before_create and after_create inside the same transaction so that it can roll back when the exception is raised from any of these callbacks. after_commit is executed after the transaction is committed to the database. So if you have calls like

class MyModel <ApplicationRecord
  before_create :fetch_data_from_server

  def fetch_data_from_server
    self.attribute = SlowApi.new(self).call
  end
end

The transaction will remain open for long time and that will cause locks and potentially deadlocks on tables.

Solutions:

1. Make network calls before transaction begins, ie before the save method is called

2. Make the API call in after commit callback

1. Scheduling Background jobs from after_create/before_create callbacks

This is very similar to the previous point, as the backgrounds jobs typically use another data store like Redis and the communication will be over the network. This may not be significant on a small scale if the Redis is in the same infra and network latency is not significant. But any performance degradation on redis infra will cause a sudden spike in long transactions and can potentially cause cascading failures on the infrastructure.

Solutions:

1. Schedule background jobs from after_commit callback
2. Move job scheduling out of rails callbacks

1. Nested transactions

When you have nested services you might end up with nested transactions as well.

class OrderCreator
  def call
    Order.transaction do
      adjust_currency_conversion
      ...
      InventoryUpdateService.new.call
    end
  end
end

class InventoryUpdateService
  def call
   Inventory.transaction(require_new: true) do
     deduct_inventory_from_primary_store
     run_inventory_sync
   end
  end
end

This can cause performance bottlenecks due to the SAVEPOINT behaviour, Skipping it here as Gilab wrote a great blog on this here

1. Last but the not least is about adding transaction block where it is not necessary, for instance if there won’t any data corruption if the operations ran individually, or the impact is very minimal which gets fixed on background worker retry, then avoid using database transactions

Also checkout these two great gems which allows us to reduce transaction burdern on database.

1. after_commit_everywhere
2. isolator

]]> Aboobacker M K https://ravidwivedi.in/posts/fsci-doing-ama-on-reddit/ https://ravidwivedi.in/posts/fsci-doing-ama-on-reddit/ Fri, 08 Apr 2022 20:31:36 GMT Free Software Community of India will be doing an ‘Ask Me Anything’ session on r/India subreddit. One of the moderators of the r/India reached out to us via filling the contact form on the website as well as in our chat group. As soon as I saw that the person filled the form, I replied it by email. It took some time for the community to arrive at a date and time for conducting the AMA, but now it is finalized to be 12 April 2022 17:00 IST.

Akshay created a Reddit account for FSCI and shared the credentials. Me, Praveen, Akshay and Arya will be answering the AMA. Hopefully, this will raise awareness about Free Software in India and the existence of our community. Our community is always in the ‘Ask us Anything’ mode because of our welcoming nature and our active lookout for new people to get involved. So if you miss the AMA due to some reason, then you can just ask us in our chat group. AMA is just one more way to ask us questions.

So, tune into r/India subreddit on 12 April at 5 PM IST. Ask me anything. See you there.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/selfish-gene/ https://ravidwivedi.in/posts/selfish-gene/ Fri, 08 Apr 2022 06:09:20 GMT Yesterday I finished reading The Selfish Gene by Richard Dawkins. The book reminds me of Daniel Kahneman’s Thinking Fast and Slow, which explores how the human brain makes decisions. In contrast, The Selfish Gene puts forward how genes make decisions and manipulate our decisions for their own benefit, keeping in mind that ‘selfish’ is just a metaphor and a gene has no motive or will of its own. The book presents some very great ideas by other scientists and by the author himself, and for this reason I suggest you to read it.

The larger the percentage of genes shared by the two individuals, the higher the chance of them behaving cooperatively towards each other. Each gene in a human body, and in many organisms from the animal kingdom, has a 50% chance of being in any of the person’s sibling. The same goes for the parents, i.e, Each gene in a human body has a 50% chance to be in each of its parent’s body and 50% chance of being in each offspring’s body. Interesting questions raise from the gene point of view, why then a human body care more for their offspring than its siblings? Why there is an asymmetric relationship between parental care towards the offspring and offspring care towards the parent? After all, genetically, the investment in each individual of that organism is the same.

The book also asks whether parents should invest equally in all their offsprings? How offsprings compete for resources provided by the parent? Parents also try to minimize their effort and investment in the offspring and instead try to manipulate their mate to take more responsibility for the offspring. Why females have a sudden menopause, while males do not go through such a thing?

There are even more interesting questions, such as, why we should look an organism as a whole unit? Why not consider a pack of wolves as a single unit? An organism, like me or you, is made up of complicated parts, each acting somewhat in unity. But what is so special about an organism to treat it as a single unit? Why did the genes or replicators chose to live in complicated bodies like animals?

The arguments and the selfishness surrounding them might make you cynical, but the author is optimistic that humans can be taught to be altruistic, and that gives us one more reason to teach children to be altruistic.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/kiwix-app/ https://ravidwivedi.in/posts/kiwix-app/ Thu, 31 Mar 2022 07:26:27 GMT Recently I came to know about freedom-respecting Kiwix app and I found it cool. It is a reader for offline reading knowledge sources like Wikipedia or Project Gutenberg. There are many projects which you can download for offline reading, like AskUbuntu, sections of Wikipedia– like, Chemistry, History, Sociology, etc. RationalWiki (my favorite), Ted Talks, Khan Academy, ArchWiki, etc. You can download Wikivoyage and have travel guides for offline reading, as you might not have good internet access on the go. The files are downloaded in the free format .zim which are readable by the Kiwix reader. The files are compressed, which saves disk space and internet data while downloading. Further, the zim files make it easy to index, search, and selectively decompress. It can also export files into HTML and PDF formats.

The app is available for download for all the major operating systems– Android, GNU/Linux, iOS, Windows, macOS, Raspberry Pi. In the Android app, you can download content by going to the download section and clicking on the website you would like. For Desktop, just download a zim file from the Kiwix library and open it in the Kiwix app. The project also has a nice wiki documentation.

The features make the app very usable for viewing the content offline. Getting the Wikipedia knowledge at one place, while clicking the links referring to the respective pages inside the same offline directory makes it very convenient to use. Check the features page for more details.

As much as I like offline reading due to internet being a distraction, other use cases are even better. The zim files can be downloaded and taken to places which are cut off from internet access, and spread knowledge there. Defector NGOs use Kiwix-desktop to smuggle knowledge of the outside world to North Korea, where internet is almost non-existent with extreme amounts of censorship. In Ghana, where schools have access to computers but not internet, a project downloaded Kiwix and the full English Wikipedia and Wiktionary on a portable hard drive and transferred it to school computer labs. Quoting the project, “Fewer than 10 percent of students had ever accessed Wikipedia before the training took place. Now half the students say the new resource has influenced their studies.”

You can use a Raspberry Pi to create a local server which can be used to access the downloaded content from 25 devices, which is another cool feature.

When I downloaded the 87 GB Wikipedia zim file, I ran into an immediate problem: I couldn’t copy it into my USB drive due to FAT32 filesystem which can’t store files bigger than 4 GB. So, I backed up all the data on my USB drive and used this tutorial to format my USB drive into exFAT format, which supports larger file sizes.

A drawback in the current form is that incrememtal updates are not available, which means if you download Wikipedia on 1 Feb 2022, and now you want the changes till date, then you cannot just update the changes, you need to download from scratch. The project is working on such features.

I already started using Kiwix app almost exclusively for offline reading. Try it and I am sure you will find something useful for you.

The project relies on user donations, so I would like to urge you to support the Kiwix project by making donations.

Finally, I leave you with some more screenshots of Kiwix. Enjoy :)

]]> Ravi Dwivedi https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojure/index.html https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojure/index.html Fri, 25 Mar 2022 00:00:00 GMT Aditya Athalye clojure functional_programming howto riff https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-1-doug-mcilroys-pipeline/index.html https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-1-doug-mcilroys-pipeline/index.html Fri, 11 Mar 2022 00:00:00 GMT Aditya Athalye bash unix functional_programming architecture https://saket-choudhary.me/blog/2022/03/10/fast-sparsespearman/ https://saket-choudhary.me/blog/2022/03/10/fast-sparsespearman/ Wed, 09 Mar 2022 18:30:00 GMT The previous post formulated a solution for calculating spearman correlation on sparse matrices. The trick was simple - we exploited the sparsity structure in our matrix to prevent it from being converted to a dense form at any step. For spearman correlation, following this line of thought is tricky, because the ranks are not sparse, at least not by default. We then used the property of covariance, where adding a constant quantity to all entries of a vector (or matrix) does not change its variance (or covariance) structure to come up with a solution that works great in theory and solves our purpose of keeping the memory footprint low. A notebook with an implementation here.

However, the time benchmarks look awful - though we ended up saving memory, SparseSpearmanCor() was atleast 2 times slower than the naive approach of densifying the matrix and calculating correlation using cor(as.matrix(X.sparse), method="spearman"). This in practise defeats the motivation - we are saving memory at the cost of speed.

Solution

The costliest step in my original implementation of SparseSpearmanCor() was a simple lookup operation: which(j == column) ,

where I fetch the non-zero entries in a column for calculating the rank, and this happens for all the columns (j stores the index of columns where there are non-zero entries).

I tried other ways of making this faster, such as by using fastmatch. But the actual speedup came from a simple thought - if we care about the non zero entries, I should just deal with them separately. So instead of doing repeated lookups, I just separate the non-zero entries out, do the rank sparsification operations on them and put them back into the sparse matrix. I call this implementation SparseSpearmanCor2() and you can find the implementation in the notebook, but here are some comparisons with the dense approach and the previous implementation SparseSpearmanCor().

The result is a function that calculates values 10x faster than any approach on large matrices (10000 x 5000):

Figure 1. Time benchmarks.

SparseSpearmanCor2() and time benchmarks are available in this notebook.

---

]]> Saket Choudhary https://saket-choudhary.me/blog/2022/03/09/sparsespearman/ https://saket-choudhary.me/blog/2022/03/09/sparsespearman/ Tue, 08 Mar 2022 18:30:00 GMT vector which will be the input to corSparse.]]> An important property of the covariance function is that it is invariant under shifts, i.e., for any two random variables $\mathbf{X}$ and $\mathbf{Y}$ , you get the same covariance if you add constant quantitie to either $X$ or $Y$:

$\begin{aligned} \text{Cov}(\mathbf{X} + a, \mathbf{Y} + b) &= \text{Cov}(\mathbf{X}, \mathbf{Y}) \end{aligned}$

where $\text{Cov}(\mathbf{X}, \mathbf{Y}) = \mathbb{E}[(\mathbf{X} - \mathbb{E}[\mathbf{X}])(\mathbf{X} - \mathbb{E}[\mathbf{X}])]$ and $a,b$ are real valued quantities. Essentially $\text{Cov}(\mathbf{X}, \mathbf{Y})$ is a measure of the product of how much $\mathbf{X}$ and $\mathbf{Y}$ are deviating from their respective means so adding a constant does not change anything (because the deviation from the mean remains the same).

Two commonly used correlations are pearson and spearman . A pearson correlation is essentially a normalized measure of covariance, which tries to measure how “linearly dependent” are $\textit{X}$ and $\textit{Y}$:

$\begin{aligned} \text{Cor}(\mathbf{X} + a, \mathbf{Y} + b) &= \frac{\text{Cov}(\mathbf{X}, \mathbf{Y})}{\sigma_X \sigma_Y},\\ \sigma^2_X &= \mathbb{E}[(X-\mathbb{E}[X])^2),\\ \sigma^2_Y &= \mathbb{E}[(Y-\mathbb{E}[Y])^2).\\ \end{aligned}$

The spearman correlation on the other hand asseses if the relationship between $\textit{X,Y}$ is monotonic (either increasing or decreasing). It is equivalent to running pearson correlation between the ranks of values in $X$ and $Y$ instead of the actual value themselves. So it essentially asks if $X$ is increasing (decreasing) would values in $Y$ would be increasing (decreasing) as well? A perfect score of 1 (-1) would result in a yes (no). Both the types of correlation are often employed in genomics to assess relationship between two variables of interest.

One particular context, where correlations are employed is in multi-omics experiments, say where we are profiling RNA and open chromatin regions (ATAC) in the same cells. For example, a recent study used correlations to find potential gene-enhancer links (Ma et al., 2020). The idea is simple: we have a bunch of cells in which we simultaneously profiled both the transcriptome (RNA) and the open chromatin regions (ATAC). We then ask, for each gene, which open chromatin regions are highly correlated (after necessary adjustment for background) to predict potential gene-enhancer links. The default correlation function in R cor(RNA, ATAC, method="pearson") or cor(RNA, ATAC, method="spearman") would ideally be sufficient to do this. Here, RNA and ATAC are vectors of equal length with entries summarizing the transcriptome signal and ATAC signal at a gene and potential enhancer, respectively.

However, both RNA and ATAC matrices are often sparse matrices, i.e. they have lots of entries that zeroes, which are not explicitly stored to save space. The default cor() method does not work on sparse matrices. The problem here is a simple one then: convert the RNA and ATAC sparse matrices to a usual (dense) matrix using as.matrix() and run the correlation function. However, converting to denser matrix format will take loads of memory, especially if you are searching for link between 10,000 genes and say only about 5,000 potential enhancers in around 10,000 cells all at once, parallely.

Sparsity makes it easier

The solution to avoid this is rather easy and has been previously discussed for pearson correlation. A detailed description is available in the documentation of qlcMatrix::corSparse(). But in short, the idea is to utilize the sparsity in a vector and avoid doing operations that would make a sparse matrix dense. For example, the variance calulation for a sparse vect the essential idea here is that we do not want to lose the sparsity structure during our calculations. For example, for a sparse vector, if we are interested in calculating the variance $Var(X) = \mathbb{E}[(X-\mathbb{E}[X])^2]$, if we do the $X-\mathbb{E}[X]$ operation first, the sparsity structure of X is now destroyed and we land up with a dense matrix. Instead, we can use the fact that the variance can equivalentyl be written as $\text{Var}(X) = \mathbb{E}[X^2] - E[X]^2$, retaining the sparity throughout. That solves our problem of calculating pearson correlation on sparse vectors (or matrices).

The next question is then, what about sparse matrices and spearman correlation? cor(X, Y, kind="spearman") does not work for sparse matrices and we do not want to convert them to dense form. The solution is again simple, but took me a while to figure out. A naive idea would be to use the definition of spearman correlation - we calculate ranks of $X$ and $Y$ and then run it through cor() with method="spearman" as the ranks are not sparse. The problem however is again the same - the rank matrix is not sparse. But if you think about ranks in a sparse matrix, it does have some interesting properties that we can utilize to make it sparse.

We can look at a sparse vector for an example. Consider a vector y <- c(0,0,0,42,21,10) with 3 non zero entries. We will use $n_z$ to denote the number of non-zero entries in a vector. But if we know the number of non-zero entries, we also know what these ranks are going to be - they are fixed. For a vector with $n_z$ entries, the rank(ties.method="average") method will set them all to $\frac{1}{n_z}\sum_{i=1}^{n_z} i = \frac{(n_z+1)}{2}$. We also know that the lowest non-zero entry in such a vector would have a rank of $(n_z+1)$. For example, rank vector rank(y) = c(2,2,2,6,5,4) - by default the ranks of tied entries are averaged. So the rank of 0s is $\frac{1+2+3}{3}= \frac{(n_z+1)}{2}$. Our rank vector is not sparse, but we can retain its sparsity if we were to subtract $\frac{(n_z+1)}{2}$ from each of the entries. Since a shift operation will not change the (co)variance, the variance of c(0,0,0,4,3,2) which we called the “sparsified rank vector” is the same as original rank vector c(2,2,2,6,5,4). So we should aim to get our “sparsified rank” vector somehow.

The trick to arrive at “sparsified rank” vector is to use calculate ranks on the non-zero entries in our vector. We will forget about the zero entries in such a vector and only focus on the non-zero entries - they are few and it is fast to calculate ranks of just these. In this version of the vector (where there are no zeros) the lowest non-zero entry has a rank of $1$ (assuming there are no ties, but the following arguments hold without loss of generality). To arrive at the “sparsified rank” vector, we subtracted $\frac{(n_z+1)}{2}$ from the original rank vector, so the non-zero entry’s rank will now be $n_z + 1 - \frac{(n_z+1)}{2} = 1 + \frac{n_z}{2} - \frac{1}{2}$ which is equivalent to adding $\frac{(n_z-1)}{2}$ to the rank of the non-zero entries! By this way, we retain the sparsity in ranks and can then just use corSparse() to calculate pearson correlation on sparsified rank vectors, resulting in spearman correlation.

While this approach is memory efficient, it unfortunately is not always the fastest. See this notebook for some time benchmarks. I did not explicitly perform memory benchmarks.

Update: The approach is both memory efficient and fast. See an updated post and associated notebook

Example

y <- c(0,0,0,42,21,10)

rank(y) = c(2,2,2,6,5,4)

sparsified_rank(y) <- c(0,0,0,4,3,2) (Subtract $\frac{(n_z+1)}{2}=2$ from all entries to make the previous vector a sparse vector)

rank(y[y!=0]) = rank(c(42,21,10)) = c(3,2,1) .

If we now add $\frac{(n_z-1)}{2} = \frac{(3-1)}{2}$ to all the entries of the last vector, we get c(4,3,2) which are the non-zero ranks from our <code<sparisifed_rank</code> vector which will be the input to corSparse.

---

]]> Saket Choudhary https://www.evalapply.org/posts/shite-the-static-sites-from-shell-part-1/index.html https://www.evalapply.org/posts/shite-the-static-sites-from-shell-part-1/index.html Tue, 08 Mar 2022 00:00:00 GMT Aditya Athalye design websites frontend web_development bash howto whyto https://nadh.in/blog/fomo-yamo/ https://nadh.in/blog/fomo-yamo/ Wed, 02 Mar 2022 00:00:00 GMT A whole new way of seamlessly “hydrating” and building “reactive” webpages, proclaim the dozen new Javascript frameworks that offer slightly different ways of manipulating DOM; new stacks for generating static webpages from templates; new ways of deploying “no-code” apps to “serverless edges”; memory-safe languages that enable error-free programs; NoSQL databases that offer unlimited scalability; CSS frameworks that forever change how webpages are styled; new paradigms of visualizing programs as containers and not processes, container orchestration and not process management; functional programming over imperative over object oriented; “AI/ML” for whatever one pleases … magic bullets for everything.

]]> Kailash Nadh https://ravidwivedi.in/posts/indian-pirates-membership/ https://ravidwivedi.in/posts/indian-pirates-membership/ Tue, 01 Mar 2022 00:00:00 GMT I am happy to announce that I became a permanent member of Indian Pirates a few days ago. Indian Pirates is a decentralized group which aims to be a political party someday and contest elections to form a government. I hope to bring some change politically to improve our lives. The group seeks long term change which means that we are not thinking that we will change the world in day or in one year, but rather understanding the roots of the problems in society and how it is structured and solve the problems by the root itself.

It is only an announcement post and I want to keep it short. Good Bye. And see you in the next post.

]]> Ravi Dwivedi https://www.evalapply.org/posts/dismal-arithmetic-dyalog-apl-clojure/index.html https://www.evalapply.org/posts/dismal-arithmetic-dyalog-apl-clojure/index.html Fri, 25 Feb 2022 00:00:00 GMT Aditya Athalye clojure apl functional_programming riff https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-0-intro/index.html https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-0-intro/index.html Wed, 23 Feb 2022 00:00:00 GMT Aditya Athalye bash unix functional_programming architecture https://www.evalapply.org/posts/what-makes-functional-programming-systems-functional/index.html https://www.evalapply.org/posts/what-makes-functional-programming-systems-functional/index.html Tue, 22 Feb 2022 00:00:00 GMT Aditya Athalye meta functional_programming architecture systems https://www.evalapply.org/posts/dont-hurry-dont-stop-sad-version/index.html https://www.evalapply.org/posts/dont-hurry-dont-stop-sad-version/index.html Mon, 14 Feb 2022 00:00:00 GMT Aditya Athalye meta mentalhealth https://www.evalapply.org/posts/people-culture-values-strategy-technology/index.html https://www.evalapply.org/posts/people-culture-values-strategy-technology/index.html Fri, 11 Feb 2022 00:00:00 GMT Aditya Athalye organisation_design systems culture https://www.evalapply.org/posts/reader-app-pandoc-bash/index.html https://www.evalapply.org/posts/reader-app-pandoc-bash/index.html Thu, 10 Feb 2022 00:00:00 GMT Aditya Athalye functional_programming bash unix riff https://mrkaran.dev/posts/exposing-services/ https://mrkaran.dev/posts/exposing-services/ Thu, 10 Feb 2022 00:00:00 GMT :80:80" - ":443:443" networks: - public ... caddy_internal: ... ports: - "100.111.91.100:80:80" - "100.111.91.100:443:443" networks: - internal ... networks: public: name: caddy_public internal: name: caddy_internal This is where most of the magic lies. I use 2 Docker networks caddy_public and caddy_internal. Both these networks are configured as Bridge Networks. The containers connected to the same bridge network can even reach other containers using internal DNS. The published port section is the one of importance here. In the internal caddy instance, the TCP port 80 in the container is mapped to port 80 on the Docker host for connections to host IP 100.111.91.100 (this is a private IP and belongs to the CGNAT space). The same is done for caddy_public where instead of Tailscale IP, the Floating IP of the DigitalOcean droplet is used. Next comes the part where we’ll attach these networks to our applications. docker-compose by default creates a user-defined bridge network if you leave networks unspecified. However, if you want more granular control, you can specify the networks in the Compose spec itself. Here’s an example of Plausible compose spec which is exposed publicly: plausible_events_db: image: yandex/clickhouse-server:21.3.2.5 networks: - plausible plausible: image: plausible/analytics:latest networks: - web - plausible networks: web: name: caddy_public external: true plausible: name: plausible Here you can see that the ClickHouse container is only attached to the plausible network. This plausible network is scoped only to these services defined in this file. We can exec inside the caddy_public container and find out the IP of plausible and verify if the network is correctly configured and reachable: $ host plausible plausible has address 172.20.0.3 $ curl plausible:8000 You are being redirected./srv # Some notes on this setup: You’ll also note that I haven’t published the port 8000 anywhere. Publishing is only required if you want to forward the traffic from your host network to the docker network (via the bridge). But here, since both are attached to the same network (caddy_public), that is not required anymore. This also means that the only way someone can reach port 8000 on Plausible is only via the Caddy container (which is firewall restricted to Cloudflare IPs.) The DB container doesn’t need to be accessed at all from Caddy, so we’ve not attached the web network there Here’s another example of exposing an internal service, which works on the same principles: grafana: image: grafana/grafana:8.3.4 networks: - monitoring - internal networks: internal: name: caddy_internal external: true monitoring: name: monitoring Here, the Grafana container is attached to caddy_internal network. Since the caddy_internal container only publishes ports on the Tailscale IP, anyone who is not inside the Tailscale network will not be able to access this. Tailscale can do much more by setting up ACL rules per device for each user, but since I am the only user, I’ve not configured ACL rules on it yet. Hope this approach was simplistic enough. I follow this pattern across all the applications I self-host and honestly pretty happy with it Fin!]]> I’ve often been asked how to expose public and private services running on DigitalOcean droplets/RPis when self hosting apps. Most people don’t have access to a static IP for their RPis. I felt I’d summarize my approach to this in this blog post and hope it’ll be useful for others trying to do the same.

Tools I use#

• Tailscale: I love this and I’ve written about it in the past. I am one of the early adopters, using this for 2 years now and while it can be replaced with other similar Wireguard based mesh services that have popped up recently, I still like the UX of this application a lot. In simpler words, it just works!
• Caddy: I use Caddy as a webserver to host some static websites and as a reverse proxy for all the applications I’ve deployed. It makes the SSL setup a breeze, the config is nice to read as well. There’s nothing about Caddy to not like it if you’re not in the business of dealing with super high concurrent traffic on your website (in which case NGINX/HAProxy/Envoy et al might be worth looking at). I’ve never used Traefik myself, and while I know it makes the routing simple with Docker labels, I think my current approach with Caddy is also similar and easier for me to use without having to learn a new config.
• Docker: All the services are containerized and I make special use of Docker networks, which I’ll show in the post.

Setup Overview#

I use 2 instances of Caddy for my setup:

• Public: This is to reverse proxy all the public-facing websites.
• Private: This is to reverse proxy all the internal websites.

The docker-compose looks like this:

version: "3.7"

services:
  caddy_public:
    ...
    ports:
      - "<do_floating_ip>:80:80"
      - "<do_floating_ip>:443:443"
    networks:
      - public
    ...

  caddy_internal:
    ...
    ports:
      - "100.111.91.100:80:80"
      - "100.111.91.100:443:443"
    networks:
      - internal
    ...

networks:
  public:
    name: caddy_public
  internal:
    name: caddy_internal

This is where most of the magic lies. I use 2 Docker networks caddy_public and caddy_internal. Both these networks are configured as Bridge Networks. The containers connected to the same bridge network can even reach other containers using internal DNS.

The published port section is the one of importance here.

In the internal caddy instance, the TCP port 80 in the container is mapped to port 80 on the Docker host for connections to host IP 100.111.91.100 (this is a private IP and belongs to the CGNAT space). The same is done for caddy_public where instead of Tailscale IP, the Floating IP of the DigitalOcean droplet is used.

Next comes the part where we’ll attach these networks to our applications. docker-compose by default creates a user-defined bridge network if you leave networks unspecified. However, if you want more granular control, you can specify the networks in the Compose spec itself.

Here’s an example of Plausible compose spec which is exposed publicly:

  plausible_events_db:
    image: yandex/clickhouse-server:21.3.2.5
    networks:
      - plausible

  plausible:
    image: plausible/analytics:latest
    networks:
      - web
      - plausible

networks:
  web:
    name: caddy_public
    external: true
  plausible:
    name: plausible

Here you can see that the ClickHouse container is only attached to the plausible network. This plausible network is scoped only to these services defined in this file.

We can exec inside the caddy_public container and find out the IP of plausible and verify if the network is correctly configured and reachable:

$ host plausible
plausible has address 172.20.0.3
$ curl plausible:8000
<html><body>You are being <a href="/login">redirected</a>.</body></html>/srv # 

Some notes on this setup:

• You’ll also note that I haven’t published the port 8000 anywhere. Publishing is only required if you want to forward the traffic from your host network to the docker network (via the bridge). But here, since both are attached to the same network (caddy_public), that is not required anymore.
• This also means that the only way someone can reach port 8000 on Plausible is only via the Caddy container (which is firewall restricted to Cloudflare IPs.)
• The DB container doesn’t need to be accessed at all from Caddy, so we’ve not attached the web network there

Here’s another example of exposing an internal service, which works on the same principles:

  grafana:
    image: grafana/grafana:8.3.4
    networks:
      - monitoring
      - internal

networks:
  internal:
    name: caddy_internal
    external: true
  monitoring:
    name: monitoring

Here, the Grafana container is attached to caddy_internal network. Since the caddy_internal container only publishes ports on the Tailscale IP, anyone who is not inside the Tailscale network will not be able to access this. Tailscale can do much more by setting up ACL rules per device for each user, but since I am the only user, I’ve not configured ACL rules on it yet.

Hope this approach was simplistic enough. I follow this pattern across all the applications I self-host and honestly pretty happy with it

Fin!

]]> Karan Sharma https://ravidwivedi.in/posts/hindi-typing-in-debian-based-distros/ https://ravidwivedi.in/posts/hindi-typing-in-debian-based-distros/ Mon, 07 Feb 2022 00:00:00 GMT Input Devices -> Keyboard -> Layouts -> Add layout To add the layout, fill the following details: Limit selection by language: Any language Layout: Indian (They have written ‘Indian’ as a language, which should be ‘Hindi’) Variant: default Label: in Click ‘Ok’. Tip: Ctrl + Alt + K keys is the shortcut for switching the input language. That sets up typing in Hindi. Mapping of inscript Hindi keyboard Now for the problem of fonts, Raghu suggested me to download fonts-lohit-deva package which downloads Lohit Devanagari font. After downloading, run the command fc-cache -v | grep lohit to see if lohit is there. If it is there, we are good to go. In LibreOffice Impress, I changed the font to ‘Lohit Devanagari’ and the problem got fixed. At last, I made the slides and recorded the video, which you can watch here.]]> A few days ago, I was contacted by IT for Change to make a video presentation on importance of Free Software in education to teachers in Hindi. When I was making slides in LibreOffice Impress, I saw some problems with Hindi typing. First, I was not able to set the input language to Hindi and second, the fonts in LibreOffice Impress weren’t rendering correctly.

So I asked my friend Raghu for help on how to type in Hindi and how to fix the issue, who helped me. I am documenting my findings here for future reference or helping other people.

I am using PureOS with KDE right now. The same should work in other GNU/Linux distros too, at least in Debian-based ones.

To type in Hindi, go to System Settings -> Input Devices -> Keyboard -> Layouts -> Add layout

To add the layout, fill the following details:

• Limit selection by language: Any language

• Layout: Indian (They have written ‘Indian’ as a language, which should be ‘Hindi’)

• Variant: default

• Label: in

Click ‘Ok’.

Tip: Ctrl + Alt + K keys is the shortcut for switching the input language.

That sets up typing in Hindi.

Now for the problem of fonts, Raghu suggested me to download fonts-lohit-deva package which downloads Lohit Devanagari font. After downloading, run the command fc-cache -v | grep lohit to see if lohit is there. If it is there, we are good to go.

In LibreOffice Impress, I changed the font to ‘Lohit Devanagari’ and the problem got fixed.

At last, I made the slides and recorded the video, which you can watch here.

]]> Ravi Dwivedi https://www.evalapply.org/posts/how-to-give-a-conference-talk/index.html https://www.evalapply.org/posts/how-to-give-a-conference-talk/index.html Thu, 03 Feb 2022 00:00:00 GMT Aditya Athalye meta writing speaking https://www.evalapply.org/posts/software-debt/index.html https://www.evalapply.org/posts/software-debt/index.html Thu, 20 Jan 2022 00:00:00 GMT Aditya Athalye risk complexity bias systems programming architecture culture quality software_design scale https://www.evalapply.org/posts/how-to-not-die-by-a-thousand-cuts/index.html https://www.evalapply.org/posts/how-to-not-die-by-a-thousand-cuts/index.html Thu, 20 Jan 2022 00:00:00 GMT Aditya Athalye quality risk systems https://ravidwivedi.in/posts/snikket-experience/ https://ravidwivedi.in/posts/snikket-experience/ Thu, 20 Jan 2022 00:00:00 GMT What is Snikket?

Snikket is a software that can be installed on a server to self-host an xmpp server. Snikket makes some choices for the person installing it on the server and therefore, makes it easy to get the features any XMPP server would usually like to have– audio/video calls, file sharing, etc. It is very convenient to set up and saves time. The downside is that you lose control over your setup, but you can choose to install plain XMPP on the server anyway if you want more control.

When Praveen told me the idea of Snikket a few days ago, I liked the idea and so I wanted to try and self-host it.

Setting up on the server

Sahil and I first tried to get Snikket running on a server with nginx running on it, but we failed to do so. Snikket’s guide assumes that there is no reverse proxy running on the server. Then I thought that we should try deploying Snikket on a server in which there is no other service running. Sahil allowed access to a server that he was abandoning anyway. He gave a fresh Debian installed on the server. After that, we followed the Snikket self-hosting guide which was very simple and within a few minutes, the server was up and running.

Experience as a user

I created an admin invite link for myself. Then I scanned the QR code in the invite link using the Snikket app. The app asked me to set a username and password, after which my account got created and ready to go. Every user on Snikket needs an invite to join. The admin creates an invite link and shares with the person. The link gives various ways of joining and links to app stores to download the app as well. Then I created an invite link for Sahil. He joined using the Conversations app, which shows that you can use any xmpp app to accept the invite and use your account. we tried texting, file sharing, audio/video calls and it was working very well. Snikket gave 95% compliance to my server out of the box, which usually takes a lot of work in plain XMPP, but Snikket made it easy. On this occassion, the founder of Snikket project, Matt replied to me on Mastodon reminding us not to take compliance measure too seriously:

While generating the invite link, the admin can choose the circle of the person they are inviting. People joining the same circle are automatically added as contacts. They don’t need to add manually. For example, you might have a circle called Family. So, everyone in the Family circle will know each other. You might have a different circle called ‘College’ and so on.

What I like about the Snikket is that it is very good quality out of the box. The audio/video call quality is superb, file sharing works like a breeze.

Further goals

The next step is to configure Snikket on a server running a reverse proxy like nginx. Me and Sahil tried but didn’t get success. If that is successful, I can run the server in long-term. For now, it is only experimental and temporary.

Goodbye for now.

Update on May 15 2022

After a week of writing this post, I could set up Snikket on nginx. It’s been almost 5 months now and the service is very smooth. I only had to update once and that was very easy. I faced no issues as of now on the server side. The uptime has been 100%.

I found Snikket project chatroom as very helping and welcoming. They try their best to help newbies in self-hosting, which usually takes a lot of patience.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/how-i-got-into-free-software/ https://ravidwivedi.in/posts/how-i-got-into-free-software/ Fri, 07 Jan 2022 00:00:00 GMT Before I forget my journey, I hastily record it here.

It is a long series of triggers and questions raised in my mind after I met Richard Stallman in 2016 which were responsible for this.

I thought on the issues that he raised but didn’t act for years. Occasionally, thoughts about WhatsApp spying on me came to mind. Thoughts about Google recording my whole life. I told one of my friends about Facebook surveillance, and they said they have nothing to hide. Another friend I told about Uber tracking us. Another friend I told about the music app tracking us, and they responded by saying that they used it for their benefit by getting recommendations related to their music taste. All this time, I didn’t think hard enough. I merely knew about tracking, and I gave up my rights in exchange for convenience.

But many of the times, there were search results logged by Google which I didn’t want to share with them. The question was what should I do? What should anyone do to browse the web who doesn’t want to sign some arbitrary terms and conditions set up by a company who is not even accountable for their actions. So, I used a proprietary VPN app from the Play Store to “protect” myself. But Google still showed the search results in my profile. Now I really don’t want to tell Google where I am or what I search for, but on the other hand, services like Google search and Google Maps do give convenience. Also, not having a single person sharing this view in my immediate friends circle also discouraged me constantly, even if I wanted to do something about it.

In this time period, I also saw some ads by DuckDuckGo on Quora which constantly reminded me about me being pissed off with Google and their surveillance. Thanks, DuckDuckGo :)

Now, in June 2020, my master’s degree got over, and I got some time to think about the issue. I took the challenge. And as it was, it started with this challenge. I thought switching a little software or using Tor or Brave will give me privacy. I thought I will probably do it in 2-3 days. As it turned out, that thought was really stupid! The so called Dunning-Kruger effect was at work, which says that beginners in a field of study tend to overestimate their knowledge.

Initially it started as a challenge and not a quest to change the world or anything. Then I stared searching in DuckDuckGo and felt happy about it. It felt cool because nobody I knew had DuckDuckGo as their default search engine. I also used to search Reddit for information on the issue. I came to know about browser fingerprinting technique used by websites like Google to track users. As depressing as it was, using a VPN does not magically make the user invisible and there were so many complexities involved in making decisions on how to be private or anonymous.

I came across Glenn Greenwald’s book ‘No Place To Hide’ and I read it fully which was about Snowden’s revelations of massive surveillance by NSA. I felt so much for Snowden. His courage also inspired me. He risked his life to tell the world about NSA’s secret spying programs. Whenever I talked to someone about the surveillance, they seemed reluctant to change or even admit that there is some threat. And I thought that this person named Snowden risked his whole life by standing for us against the powerful US Government, but we cannot even compromise on a few conveniences for our privacy.

I didn’t had a framework to think on how to go about this. I wanted digital privacy but it is a complicated thing. I didn’t even think at that time on a threat model. I also used to think ‘What if the replacement services or software are spying on me?’. How do I verify?

This is where I took a page from Richard Stallman’s book, perhaps, I took many pages. It occured to me that ‘if we cannot even inspect what the software does, how can we be sure that the software respects privacy?’ I realized that users do not control a nonfree software. I took some time but I got convinced about the free software being necessary but not sufficient for privacy. So, the replacements I started using, like VLC Media Player or other software, were free software and this was a big difference. Now I had an answer to why use this software and how is it different. At least I was not trusting blindly on the software. I also realized that all the freedoms of free software are important and privacy is not the only issue.

At the time, I didn’t had any idea on how to think about messenger apps or search engines or anything server-based which you do not even install in your own computer.

By September 2020, I already boycotted most of the software/services by Google, Amazon, Microsoft, Facebook and started using Free Software as much as possible. Back then, I had a Macbook so boycotting Apple would take more time. In September 2020, I came to know about FSF India and immediately wrote a volunteer request to FSF India. I wanted to meet like-minded people from India. Till that point, I was doing things all alone, by myself. After a few days, Pirate Praveen replied my volunteer mail. So that is how I joined the community and get in touch with others. I also participated in Software Freedom Camp 2020 where I got to meet more people from the free software movement.

In the camp, I met many people. Sahil and Arun were in my group and we had meetings late night to discuss about privacy and teaching each other. In the later stage, I met Akhil, Anupa, Bady, Akshay. And then Karthik who taught some Hugo and command line to me.

After interaction with the community, I realized that centralization of services is another problem and decentralization, federation of services is necessary for user’s freedom, in addition to free software. That enhanced my thinking, and now I could think on ‘Freedom in the cloud’. I also realized how free software community works in practice. Earlier, I only knew the philosophy of free software.

Fast-forward to April 2021, I left FSF India due to their support for Richard Stallman at the event of him being re-elected as a board member of FSF. After that, I focused on campaigning for free software on behalf of Free Software Community of India(FSCI). I became an activist by this time. I think the main reason I became an activist is that you cannot use Free Software or have privacy/anonymity in isolation. Whatever the majority of people use is usually forced upon the minority. Free Software users who care for philosophy of free software are in minority, and therefore we have to constantly fight, for example, against the social pressure of using WhatsApp. I also realized that the real way to solve the problem is raising awareness on the issue and help people in switching to Free Software. User boycott of nonfree software is a powerful way to defeat the companies who exploit users for using their services.

Naturally, I wanted to switch to fully free software, which means not using any proprietary software. I bought a Liberated Computer which can run exclusively free software, thanks to Abhas. Abhas also taught me how to install a custom ROM in the phone so that I can get rid of proprietary apps. Now I could do my computing without giving away freedom, which is a unique accomplishment in today’s world. This way I switched to free software and my Macbook passed on to someone else who seems happy to be imprisoned by it(but I am not happy that I did this to someone).

Thanks to all the people who made it possible. First of all, huge thanks to my parents for cooperating (to an extent). Without them supporting me, it wouldn’t have been possible. There are many people who took time and effort to teach me many things. Also, there were already technical solutions for freedom. Free Software movement has existed for long. Thanks to all the people for their contributions to the software I rely on. Thanks to the Free Software Community.

I would like to take the opportunity to say that we do not really lack in technical solutions for these problems, but rather the willpower of people to use them.

In the end, I would like to say that the journey has not ended, it has only started.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/quicksy-app/ https://ravidwivedi.in/posts/quicksy-app/ Fri, 31 Dec 2021 00:00:00 GMT What is Quicksy?

Quicksy is a chatting app for Android which is available on Google Play Store and F-Droid.

Quicksy offers the same convenience, i.e., signup using phone number, as WhatsApp, Telegram and Signal. Signal respects users freedom and privacy, but Quicksy, in addition, is interoperable while Signal is not. iOS does not have Quicksy but Monal or Siskin IM on ioS are compatible with Quicksy.

Here is a video tutorial to get started with Quicksy.

List of a few features of Quicksy:

• Available on F-Droid and Play Store.

• Quicksy supports encryption by default and due to software code being available, we can verify that it does encrypt messages (without any cheating!). Encryption means only the users exchanging messages can read them. Quicksy respects users privacy in this way. The Quicksy server cannot read your messages.

• Very good video/audio call support.

• Sharing of photos/videos/voice message, in encrypted form.

• If any of your contacts register with Quicksy service, then Quicksy automatically detects and shows them in your Quicksy contact list.

• There are many apps and services compatible with Quicksy, like Conversations, Blabber, Monal etc and therefore, you are not locked into one service provider, unlike WhatsApp, Telegram, Signal.

• One server goes down does not imply that the whole XMPP communications goes down. This is due to the decentralized nature of XMPP which Quicksy is a part of.

• Easy migration: If Quicksy does something in future that you do not agree with, you can just use another XMPP service and still be able to talk to your contacts.

Why is Quicksy different?

Popular messenger apps like WhatsApp, Telegram and Signal have problems that users do not control them. For details, please check this article. WhatsApp uses the control over its users to put them under surveillance. Telegram and Signal are free software but centralized. WhatsApp is nonfree software as well as centralized. Centralized services are easily suspectible to backdoors, can be compromised later on due to change of privacy policy or terms and conditions, are easily for government to ban etc. The owner of the platform dictates the decisions of a centralized system, so even if it is good for now, it cannot be trusted to remain good forever.

Quicksy is free/swatantra/mukt/libre software (‘free’ as in freedom) and therefore users can inspect, modify and share the software. If users want to add a feature, fix a problem with the software, they don’t have to beg the developer. In other words, the software is under user’s control.

Quicksy is a part of XMPP protocol which decentralized and federated. To understand federation, we give example of mobile telecom operators. A person using a BSNL SIM can talk to a user having Vodafone SIM, by call and SMS. WhatsApp, Telegram and Signal do not give you that freedom. Your all the contacts need to be registered with the same service provider. These apps have a fundamental problem of vendor lock-in – that if you uninstall WhatsApp, for example, you lose all your contacts. With federation, it is like changing SIM. If you have a problem with a telecom operator and you buy a new SIM of another company, you do not lose all your contacts. You can still call and message your earlier contacts. This way we can control our communications rather than relying on a single entity for our communications.

Let’s choose Quicksy for a free society rather than being locked by WhatsApp, Telegram or Signal.

A word for Free Software Community

Dear Free Software Community, let’s raise awareness about Quicksy. We don’t have to recommend our geeky solutions to everyone. If some people switch to free software, decentralized and federated app like Quicksy, that is better compared to them finding xmpp/matrix/IRC difficult to use and not using it. We need to work to advertise this option and raise awareness about it.

Further Reading:

• Introduction to Free Software.

• Why Free Software and decentralization are necessary for privacy.

• WhatsApp is malware.

• What Does The Facebook Outage Teach Us.

• Choosing a privacy-respecting chatting app.

• Reclaim privacy in instant messaging with Free Software and choice of service providers.

]]> Ravi Dwivedi https://mrkaran.dev/posts/clickhouse-replication/ https://mrkaran.dev/posts/clickhouse-replication/ Fri, 17 Dec 2021 02:40:55 GMT section is present inside the config. Here’s a sample config: 9181 ${SERVER_ID} /var/lib/clickhouse/coordination/log /var/lib/clickhouse/coordination/snapshots 10000 30000 trace 1 clickhouse-blue-1 9234 2 clickhouse-blue-2 9234 3 clickhouse-green-1 9234 There are some other configs required for Clickhouse to discover other nodes and enable replication. You can find a working example in this repo. Verifying Cluster State# Once the containers are configured and running, we can verify if the replication is working as intended: Let’s first check if the keeper daemon is running by: echo ruok | nc 127.0.0.1 9181 imok ruok is a part of Four Letter Commands that are mostly used to diagnose Keeper’s client/server. To ensure that clickhouse-server is aware of the keeper cluster, we can query the system.zookeeper table: SELECT * FROM system.zookeeper WHERE path = '/' FORMAT Vertical Query id: 287d3c2d-b93f-4d48-b335-6df2f89a8ab3 Row 1: ────── name: clickhouse value: czxid: 3 mzxid: 3 ctime: 2021-12-17 09:11:05 mtime: 2021-12-17 09:11:05 version: 0 cversion: 1 aversion: 0 ephemeralOwner: 0 dataLength: 0 numChildren: 1 pzxid: 4 path: / If you don’t see any results in the system.zookeeper table, then re-check if zookeeper section is present inside the config. This config tells ClickHouse how to discover keeper nodes. We can also see if our cluster is configured correctly with: SELECT host_name, host_address, replica_num FROM system.clusters WHERE cluster = 'events' Query id: a4bacfa1-d3aa-482f-b8b2-30b05442a173 ┌─host_name──────────┬─host_address─┬─replica_num─┐ │ clickhouse-blue-1 │ 172.19.0.5 │ 1 │ │ clickhouse-blue-2 │ 172.19.0.3 │ 2 │ │ clickhouse-green-1 │ 172.19.0.4 │ 1 │ │ clickhouse-green-2 │ 172.19.0.2 │ 2 │ └────────────────────┴──────────────┴─────────────┘ (Here events is our cluster name specified in the remote_servers section of the config.) Inserting Sample Data# Let’s create a DB and add some data to the DB. We need to ensure that our data is split across shards and we can query all shards using a central view. Cluster Schema# CREATE DATABASE app ON CLUSTER 'events'; CREATE TABLE app.events_local ON CLUSTER '{cluster}' ( time DateTime, event_id Int32, uuid UUID ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{cluster}/{shard}/{table}', '{replica}') PARTITION BY toYYYYMM(time) ORDER BY (event_id); CREATE TABLE app.events_main ON CLUSTER '{cluster}' AS app.events_local ENGINE = Distributed('{cluster}', app, events_local, rand()); What’s happening here: We’ll create a sample database app with a single table events_local. We’re using ReplicatedMergeTree here as that tells ClickHouse to automatically replicate the data inside the table when it’s inserted. Properties like cluster/shard/replica are automatically populated from the server’s macros. It’s a handy feature that allows you to execute this command just once on the cluster and it automatically populates the relevant config in each server. Finally we create a Distributed table to perform our INSERT/SELECT operations in a central place. It’s possible to manually insert data to particular replicas, but since Clickhouse supports load balancing across shards, it’s preferred to create a table with a Distributed engine and let that happen automatically. Write operations can be sharded based on a particular column name, but here we are simply using the rand() function, which splits the write randomly to different shards. All read operations are parallelized and Clickhouse selects one replica from each shard to query the data from. We use ON CLUSTER keyword to indicate that this query has to be run on all servers which are part of the cluster events. It’s now time to insert some random data, for which you can use this command: INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4()); We can now query the table and see if our records are present: SELECT * FROM app.events_main ┌────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 09:36:17 │ 1888949839 │ 3c33305e-8ea0-4ac1-a07a-667465ec9a85 │ └─────────────────────┴────────────┴──────────────────────────────────────┘ ┌────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 09:36:16 │ -689113926 │ 2a944d71-73f3-4491-b851-4f0e6f296e44 │ └─────────────────────┴────────────┴──────────────────────────────────────┘ ┌────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 09:36:15 │ 415899002 │ 5980299f-1594-4c17-8eb5-e512f15ecf34 │ └─────────────────────┴───────────┴──────────────────────────────────────┘ ┌────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 09:36:16 │ 1303963476 │ 0b27d5bd-6315-4937-82a4-18199e5eebb7 │ └─────────────────────┴────────────┴──────────────────────────────────────┘ Now, you must be wondering how to check if the data is replicated and how the data is distributed across our shards. For that, we can use the handy remote() function: SELECT * FROM ( SELECT hostName(), * FROM remote('172.20.0.2', 'app', 'events_local') UNION ALL SELECT hostName(), * FROM remote('172.20.0.3', 'app', 'events_local') UNION ALL SELECT hostName(), * FROM remote('172.20.0.4', 'app', 'events_local') UNION ALL SELECT hostName(), * FROM remote('172.20.0.5', 'app', 'events_local') ) Query id: 34a81447-a31b-4915-9bd8-7a6e17bb0860 ┌─hostName()────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-blue-2 │ 2021-12-17 09:36:17 │ 1888949839 │ 3c33305e-8ea0-4ac1-a07a-667465ec9a85 │ └───────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-blue-1 │ 2021-12-17 09:36:17 │ 1888949839 │ 3c33305e-8ea0-4ac1-a07a-667465ec9a85 │ └───────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-2 │ 2021-12-17 09:36:15 │ 415899002 │ 5980299f-1594-4c17-8eb5-e512f15ecf34 │ └────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-2 │ 2021-12-17 09:36:16 │ -689113926 │ 2a944d71-73f3-4491-b851-4f0e6f296e44 │ └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-2 │ 2021-12-17 09:36:16 │ 1303963476 │ 0b27d5bd-6315-4937-82a4-18199e5eebb7 │ └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-1 │ 2021-12-17 09:36:16 │ 1303963476 │ 0b27d5bd-6315-4937-82a4-18199e5eebb7 │ └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-1 │ 2021-12-17 09:36:15 │ 415899002 │ 5980299f-1594-4c17-8eb5-e512f15ecf34 │ └────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-1 │ 2021-12-17 09:36:16 │ -689113926 │ 2a944d71-73f3-4491-b851-4f0e6f296e44 │ └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ Perfect! We can see our data is sharded as some parts of it exists in green and blue. We can also see that for each record, we have 2 entries thus confirming that ReplicatedMergeTree is doing its job! Additional Scenarios# All is well and good so far, but to ensure that our cluster setup is resilient we need to introduce our cluster to some Non-Happy scenarios. Stop a replica node# ❯ docker-compose stop clickhouse-green-2 Stopping clickhouse-green-2 ... done Now, let’s query our records: SELECT count(*) FROM app.events_main ┌─count()─┐ │ 4 │ └─────────┘ That seems right. We’re able to access all our data with just one replica down. Let’s try inserting data and bring back the replica to see if it got automatically replicated or not: INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4()); Ok. SELECT * FROM app.events_main ORDER BY time DESC LIMIT 1 ┌────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 10:09:10 │ 375826335 │ 8dbbc9cf-8f00-4cc3-a4fb-0f17a68340e5 │ └─────────────────────┴───────────┴──────────────────────────────────────┘ Let’s start the replica again: ❯ docker-compose start clickhouse-green-2 Starting clickhouse-green-2 ... done On querying the replica to see if it has the data: SELECT hostName(), * FROM remote('172.20.0.5', 'app', 'events_local') ORDER BY time DESC LIMIT 1 Query id: 422041dc-afe8-4f28-9659-6d58726f8c90 ┌─hostName()─────────┬────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-2 │ 2021-12-17 10:09:10 │ 375826335 │ 8dbbc9cf-8f00-4cc3-a4fb-0f17a68340e5 │ └────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘ Perfect! The record 375826335 automatically got replicated once the replica was healthy. Stop a Keeper Node# We’ll stop a server instance that is running the clickhouse-keeper process. By doing this, we’ll also be killing a replica, but that is okay. ❯ docker-compose stop clickhouse-blue-2 Stopping clickhouse-blue-2 ... done Let’s insert some data: INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4()); Ok We can check the server information of the other 2 keeper nodes: $ echo stat | nc 127.0.0.1 9181 | grep Mode Mode: follower $ echo stat | nc 127.0.0.1 9183 | grep Mode Mode: leader So, one of the keeper nodes has elected itself to be the leader and we have no problems in the setup so far. However, if we stop another keeper node, then there will be only keeper node remaining in the setup and to avoid a Split Brain issue, it won’t be able to elect itself as the leader. What happens then? Only one way to find out: ❯ docker-compose stop clickhouse-blue-1 Stopping clickhouse-blue-1 ... done INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4()); Received exception from server (version 21.12.2): Code: 242. DB::Exception: Received from localhost:9000. DB::Exception: Table is in readonly mode (zookeeper path: /clickhouse/tables/events/green/table). (TABLE_IS_READ_ONLY) Ah! So, we can still query for the data (which will be incomplete since blue shard is completely down), but we cannot insert any new data at all. We can even verify this by querying for the health of the keeper node: $ echo mntr | nc localhost 9183 This instance is not currently serving requests% Add a new shard# Alright, so our tests so far have been quite good and show that the cluster is resilient to failures as long as there exists a keeper node running as leader mode in the quorum. Now, let’s see how to add a new shard. We’ll extend our docker-compose.yml to add a new orange shard: clickhouse-orange-1: <<: *clickhouse-defaults container_name: clickhouse-orange-1 hostname: clickhouse-orange-1 ports: - 9004:9000 - 8127:8123 - 9185:9181 volumes: - type: volume source: ch-orange-1-data target: /var/lib/clickhouse - "./configs/gen/clickhouse-orange-1:/etc/clickhouse-server/config.d/" Inside our remote_servers.xml, we’ll add the orange shard as well: true clickhouse-orange-1 9000 That’s pretty much it. Let’s start the new replica: docker-compose up Let’s insert some data and query if the shard is getting data or not: SELECT hostName(), * FROM remote('172.20.0.2', 'app', 'events_local') Received exception from server (version 21.12.2): Code: 519. DB::Exception: Received from localhost:9000. DB::Exception: All attempts to get table structure failed. Log: Code: 279. DB::NetException: All connection tries failed. Log: There is no table `app`.`events_local` on server: 172.20.0.2:9000 . (ALL_CONNECTION_TRIES_FAILED) (version 21.12.2.17 (official build)) . (NO_REMOTE_SHARD_AVAILABLE) Houston, we have a problem. > There is no table `app`.`events_local` on server: 172.20.0.2:9000 This is mentioned in the ClickHouse docs on Replication: CREATE, DROP, ATTACH, DETACH and RENAME queries are executed on a single server and are not replicated: This means that although we’d run CREATE DATABASE and CREATE TABLE commands using ON CLUSTER which executes on all servers, but since the orange-1 node is introduced after we ran that command, we need to manually create the DB and Table here. We have to execute the below commands inside the orange-1 replica: CREATE DATABASE app; CREATE TABLE app.events_local ( time DateTime, event_id Int32, uuid UUID ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{cluster}/{shard}/{table}', '{replica}') PARTITION BY toYYYYMM(time) ORDER BY (event_id); That’s all that is required. Adding a new replica is also the same process. Summary# Hope this tutorial helped you figure out how to use clickhouse-keeper to set up a distributed ClickHouse cluster DB. clickhouse-keeper is still a relatively new feature as the docs mention already, but given that it solves operations overhead of running a Zookeeper cluster, it’s worth checking it out. References# https://www.youtube.com/watch?v=abhcCRW09Ac Slides used in above talk: https://presentations.clickhouse.com/meetup54/keeper.pdf https://clickhouse.com/docs/en/operations/clickhouse-keeper https://github.com/ClickHouse/ClickHouse/tree/master/tests/integration/test_keeper_multinode_simple https://github.com/ClickHouse/ClickHouse/issues/2161 For the full code/config samples, you can check out the repo. Fin! Updates# If you have setup RBAC on your cluster, make sure you add and fields to the configuration.]]> ClickHouse is an extremely performant columnar DB used for fast analytical processing. ClickHouse supports data replication using Apache Zookeeper which needs to be deployed externally. While Zookeeper works well after you’ve tuned it properly, it was still an additional maintenance overhead. The good news is that you don’t have to worry about this anymore.

ClickHouse 21.12 release annoucement mentions ClickHouse Keeper as feature complete. Keeper is a replacement for Zookeeper, written in C++ and uses the RAFT algorithm for consensus across different nodes in the cluster. It also has some nice improvements over Zookeeper, such as compressed snapshots/logs of the state changes in the cluster and the ability to run it inside the server binary itself. I decided to spin up a local Clickhouse cluster and test out the new clickhouse-keeper feature.

For our local setup, we’ll set up 4 nodes. We’ll create 2 shards to distribute our data and each shard will have 2 replicas each. The setup looks something like this:

We need to run clickhouse-keeper only on 3 of these nodes to ensure a quorum. Here’s a sample docker-compose.yml to spin up these nodes as containers:

version: "3.7"

x-clickhouse-defaults: &clickhouse-defaults
  restart: unless-stopped
  image: yandex/clickhouse-server:21.12.2.17
  ulimits:
    nproc: 65535
    nofile:
      soft: 262144
      hard: 262144

services:
  clickhouse-blue-1:
    <<: *clickhouse-defaults
    container_name: clickhouse-blue-1
    hostname: clickhouse-blue-1
    ports:
      - 9000:9000
      - 8123:8123
      - 9181:9181
    volumes:
      - type: volume
        source: ch-blue-1-data
        target: /var/lib/clickhouse
      - "./configs/gen/clickhouse-blue-1:/etc/clickhouse-server/config.d/"

  clickhouse-blue-2:
    <<: *clickhouse-defaults
    container_name: clickhouse-blue-2
    hostname: clickhouse-blue-2
    ports:
      - 9001:9000
      - 8124:8123
      - 9182:9181
    volumes:
      - type: volume
        source: ch-blue-2-data
        target: /var/lib/clickhouse
      - "./configs/gen/clickhouse-blue-2:/etc/clickhouse-server/config.d/"

  clickhouse-green-1:
    <<: *clickhouse-defaults
    container_name: clickhouse-green-1
    hostname: clickhouse-green-1
    ports:
      - 9002:9000
      - 8125:8123
      - 9183:9181
    volumes:
      - type: volume
        source: ch-green-1-data
        target: /var/lib/clickhouse
      - "./configs/gen/clickhouse-green-1:/etc/clickhouse-server/config.d/"

  clickhouse-green-2:
    <<: *clickhouse-defaults
    container_name: clickhouse-green-2
    hostname: clickhouse-green-2
    ports:
      - 9003:9000
      - 8126:8123
      - 9184:9181
    volumes:
      - type: volume
        source: ch-green-2-data
        target: /var/lib/clickhouse
      - "./configs/gen/clickhouse-green-2:/etc/clickhouse-server/config.d/"

volumes:
  ch-blue-1-data:
  ch-blue-2-data:
  ch-green-1-data:
  ch-green-2-data:

clickhouse-keeper runs only if the <keeper_config> section is present inside the config. Here’s a sample config:

<clickhouse>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>${SERVER_ID}</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>

        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>trace</raft_logs_level>
        </coordination_settings>

        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse-blue-1</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse-blue-2</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>clickhouse-green-1</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>

There are some other configs required for Clickhouse to discover other nodes and enable replication. You can find a working example in this repo.

Verifying Cluster State#

Once the containers are configured and running, we can verify if the replication is working as intended:

Let’s first check if the keeper daemon is running by:

echo ruok | nc 127.0.0.1 9181
imok

ruok is a part of Four Letter Commands that are mostly used to diagnose Keeper’s client/server.

To ensure that clickhouse-server is aware of the keeper cluster, we can query the system.zookeeper table:

SELECT *
FROM system.zookeeper
WHERE path = '/'
FORMAT Vertical

Query id: 287d3c2d-b93f-4d48-b335-6df2f89a8ab3

Row 1:
──────
name:           clickhouse
value:          
czxid:          3
mzxid:          3
ctime:          2021-12-17 09:11:05
mtime:          2021-12-17 09:11:05
version:        0
cversion:       1
aversion:       0
ephemeralOwner: 0
dataLength:     0
numChildren:    1
pzxid:          4
path:           /

If you don’t see any results in the system.zookeeper table, then re-check if zookeeper section is present inside the config. This config tells ClickHouse how to discover keeper nodes.

We can also see if our cluster is configured correctly with:

SELECT
    host_name,
    host_address,
    replica_num
FROM system.clusters
WHERE cluster = 'events'

Query id: a4bacfa1-d3aa-482f-b8b2-30b05442a173

┌─host_name──────────┬─host_address─┬─replica_num─┐
│ clickhouse-blue-1  │ 172.19.0.5   │           1 │
│ clickhouse-blue-2  │ 172.19.0.3   │           2 │
│ clickhouse-green-1 │ 172.19.0.4   │           1 │
│ clickhouse-green-2 │ 172.19.0.2   │           2 │
└────────────────────┴──────────────┴─────────────┘

(Here events is our cluster name specified in the remote_servers section of the config.)

Inserting Sample Data#

Let’s create a DB and add some data to the DB. We need to ensure that our data is split across shards and we can query all shards using a central view.

Cluster Schema#

CREATE DATABASE app ON CLUSTER 'events';

CREATE TABLE app.events_local ON CLUSTER '{cluster}' (
    time DateTime,
    event_id  Int32,
    uuid UUID
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{cluster}/{shard}/{table}', '{replica}')
PARTITION BY toYYYYMM(time)
ORDER BY (event_id);

CREATE TABLE app.events_main ON CLUSTER '{cluster}' AS app.events_local
ENGINE = Distributed('{cluster}', app, events_local, rand());

What’s happening here:

1. We’ll create a sample database app with a single table events_local.
2. We’re using ReplicatedMergeTree here as that tells ClickHouse to automatically replicate the data inside the table when it’s inserted.
   • Properties like cluster/shard/replica are automatically populated from the server’s macros. It’s a handy feature that allows you to execute this command just once on the cluster and it automatically populates the relevant config in each server.
3. Finally we create a Distributed table to perform our INSERT/SELECT operations in a central place. It’s possible to manually insert data to particular replicas, but since Clickhouse supports load balancing across shards, it’s preferred to create a table with a Distributed engine and let that happen automatically.
   • Write operations can be sharded based on a particular column name, but here we are simply using the rand() function, which splits the write randomly to different shards.
   • All read operations are parallelized and Clickhouse selects one replica from each shard to query the data from.
4. We use ON CLUSTER keyword to indicate that this query has to be run on all servers which are part of the cluster events.

It’s now time to insert some random data, for which you can use this command:

INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4());

We can now query the table and see if our records are present:

SELECT *
FROM app.events_main

┌────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐
│ 2021-12-17 09:36:17 │ 1888949839 │ 3c33305e-8ea0-4ac1-a07a-667465ec9a85 │
└─────────────────────┴────────────┴──────────────────────────────────────┘
┌────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐
│ 2021-12-17 09:36:16 │ -689113926 │ 2a944d71-73f3-4491-b851-4f0e6f296e44 │
└─────────────────────┴────────────┴──────────────────────────────────────┘
┌────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐
│ 2021-12-17 09:36:15 │ 415899002 │ 5980299f-1594-4c17-8eb5-e512f15ecf34 │
└─────────────────────┴───────────┴──────────────────────────────────────┘
┌────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐
│ 2021-12-17 09:36:16 │ 1303963476 │ 0b27d5bd-6315-4937-82a4-18199e5eebb7 │
└─────────────────────┴────────────┴──────────────────────────────────────┘

Now, you must be wondering how to check if the data is replicated and how the data is distributed across our shards. For that, we can use the handy remote() function:

SELECT *
FROM
(
    SELECT
        hostName(),
        *
    FROM remote('172.20.0.2', 'app', 'events_local')
    UNION ALL
    SELECT
        hostName(),
        *
    FROM remote('172.20.0.3', 'app', 'events_local')
    UNION ALL
    SELECT
        hostName(),
        *
    FROM remote('172.20.0.4', 'app', 'events_local')
    UNION ALL
    SELECT
        hostName(),
        *
    FROM remote('172.20.0.5', 'app', 'events_local')
)

Query id: 34a81447-a31b-4915-9bd8-7a6e17bb0860

┌─hostName()────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐
│ clickhouse-blue-2 │ 2021-12-17 09:36:17 │ 1888949839 │ 3c33305e-8ea0-4ac1-a07a-667465ec9a85 │
└───────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘
┌─hostName()────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐
│ clickhouse-blue-1 │ 2021-12-17 09:36:17 │ 1888949839 │ 3c33305e-8ea0-4ac1-a07a-667465ec9a85 │
└───────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘
┌─hostName()─────────┬────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐
│ clickhouse-green-2 │ 2021-12-17 09:36:15 │ 415899002 │ 5980299f-1594-4c17-8eb5-e512f15ecf34 │
└────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘
┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐
│ clickhouse-green-2 │ 2021-12-17 09:36:16 │ -689113926 │ 2a944d71-73f3-4491-b851-4f0e6f296e44 │
└────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘
┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐
│ clickhouse-green-2 │ 2021-12-17 09:36:16 │ 1303963476 │ 0b27d5bd-6315-4937-82a4-18199e5eebb7 │
└────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘
┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐
│ clickhouse-green-1 │ 2021-12-17 09:36:16 │ 1303963476 │ 0b27d5bd-6315-4937-82a4-18199e5eebb7 │
└────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘
┌─hostName()─────────┬────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐
│ clickhouse-green-1 │ 2021-12-17 09:36:15 │ 415899002 │ 5980299f-1594-4c17-8eb5-e512f15ecf34 │
└────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘
┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐
│ clickhouse-green-1 │ 2021-12-17 09:36:16 │ -689113926 │ 2a944d71-73f3-4491-b851-4f0e6f296e44 │
└────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘

Perfect! We can see our data is sharded as some parts of it exists in green and blue. We can also see that for each record, we have 2 entries thus confirming that ReplicatedMergeTree is doing its job!

Additional Scenarios#

All is well and good so far, but to ensure that our cluster setup is resilient we need to introduce our cluster to some Non-Happy scenarios.

Stop a replica node#

❯ docker-compose stop clickhouse-green-2 
Stopping clickhouse-green-2 ... done

Now, let’s query our records:

SELECT count(*)
FROM app.events_main

┌─count()─┐
│       4 │
└─────────┘

That seems right. We’re able to access all our data with just one replica down.

Let’s try inserting data and bring back the replica to see if it got automatically replicated or not:

INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4());

Ok.

SELECT *
FROM app.events_main
ORDER BY time DESC
LIMIT 1

┌────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐
│ 2021-12-17 10:09:10 │ 375826335 │ 8dbbc9cf-8f00-4cc3-a4fb-0f17a68340e5 │
└─────────────────────┴───────────┴──────────────────────────────────────┘

Let’s start the replica again:

❯ docker-compose start clickhouse-green-2
Starting clickhouse-green-2 ... done

On querying the replica to see if it has the data:

SELECT
    hostName(),
    *
FROM remote('172.20.0.5', 'app', 'events_local')
ORDER BY time DESC
LIMIT 1

Query id: 422041dc-afe8-4f28-9659-6d58726f8c90

┌─hostName()─────────┬────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐
│ clickhouse-green-2 │ 2021-12-17 10:09:10 │ 375826335 │ 8dbbc9cf-8f00-4cc3-a4fb-0f17a68340e5 │
└────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘

Perfect! The record 375826335 automatically got replicated once the replica was healthy.

Stop a Keeper Node#

We’ll stop a server instance that is running the clickhouse-keeper process. By doing this, we’ll also be killing a replica, but that is okay.

❯ docker-compose stop clickhouse-blue-2
Stopping clickhouse-blue-2 ... done

Let’s insert some data:

INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4());

Ok

We can check the server information of the other 2 keeper nodes:

$ echo stat | nc 127.0.0.1 9181 | grep Mode
Mode: follower
$ echo stat | nc 127.0.0.1 9183 | grep Mode
Mode: leader

So, one of the keeper nodes has elected itself to be the leader and we have no problems in the setup so far. However, if we stop another keeper node, then there will be only keeper node remaining in the setup and to avoid a Split Brain issue, it won’t be able to elect itself as the leader.

What happens then? Only one way to find out:

❯ docker-compose stop clickhouse-blue-1
Stopping clickhouse-blue-1 ... done

INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4());

Received exception from server (version 21.12.2):
Code: 242. DB::Exception: Received from localhost:9000. DB::Exception: Table is in readonly mode (zookeeper path: /clickhouse/tables/events/green/table). (TABLE_IS_READ_ONLY)

Ah! So, we can still query for the data (which will be incomplete since blue shard is completely down), but we cannot insert any new data at all. We can even verify this by querying for the health of the keeper node:

$ echo mntr | nc localhost 9183            
This instance is not currently serving requests%                                                                                                                              

Add a new shard#

Alright, so our tests so far have been quite good and show that the cluster is resilient to failures as long as there exists a keeper node running as leader mode in the quorum. Now, let’s see how to add a new shard. We’ll extend our docker-compose.yml to add a new orange shard:

  clickhouse-orange-1:
    <<: *clickhouse-defaults
    container_name: clickhouse-orange-1
    hostname: clickhouse-orange-1
    ports:
      - 9004:9000
      - 8127:8123
      - 9185:9181
    volumes:
      - type: volume
        source: ch-orange-1-data
        target: /var/lib/clickhouse
      - "./configs/gen/clickhouse-orange-1:/etc/clickhouse-server/config.d/"

Inside our remote_servers.xml, we’ll add the orange shard as well:

 <shard>
     <internal_replication>true</internal_replication>
     <replica>
         <host>clickhouse-orange-1</host>
         <port>9000</port>
     </replica>
 </shard>

That’s pretty much it. Let’s start the new replica:

docker-compose up

Let’s insert some data and query if the shard is getting data or not:

SELECT
    hostName(),
    *
FROM remote('172.20.0.2', 'app', 'events_local')

Received exception from server (version 21.12.2):
Code: 519. DB::Exception: Received from localhost:9000. DB::Exception: All attempts to get table structure failed. Log: 

Code: 279. DB::NetException: All connection tries failed. Log: 

There is no table `app`.`events_local` on server: 172.20.0.2:9000

. (ALL_CONNECTION_TRIES_FAILED) (version 21.12.2.17 (official build))

. (NO_REMOTE_SHARD_AVAILABLE)

Houston, we have a problem.

> There is no table `app`.`events_local` on server: 172.20.0.2:9000

This is mentioned in the ClickHouse docs on Replication:

CREATE, DROP, ATTACH, DETACH and RENAME queries are executed on a single server and are not replicated:

This means that although we’d run CREATE DATABASE and CREATE TABLE commands using ON CLUSTER which executes on all servers, but since the orange-1 node is introduced after we ran that command, we need to manually create the DB and Table here. We have to execute the below commands inside the orange-1 replica:

CREATE DATABASE app;

CREATE TABLE app.events_local (
    time DateTime,
    event_id  Int32,
    uuid UUID
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{cluster}/{shard}/{table}', '{replica}')
PARTITION BY toYYYYMM(time)
ORDER BY (event_id);

That’s all that is required. Adding a new replica is also the same process.

Summary#

Hope this tutorial helped you figure out how to use clickhouse-keeper to set up a distributed ClickHouse cluster DB. clickhouse-keeper is still a relatively new feature as the docs mention already, but given that it solves operations overhead of running a Zookeeper cluster, it’s worth checking it out.

References#

• https://www.youtube.com/watch?v=abhcCRW09Ac
  • Slides used in above talk: https://presentations.clickhouse.com/meetup54/keeper.pdf
• https://clickhouse.com/docs/en/operations/clickhouse-keeper
• https://github.com/ClickHouse/ClickHouse/tree/master/tests/integration/test_keeper_multinode_simple
• https://github.com/ClickHouse/ClickHouse/issues/2161

For the full code/config samples, you can check out the repo.

Fin!

Updates#

• If you have setup RBAC on your cluster, make sure you add <user> and <password> fields to the <remote_server> configuration.

]]> Karan Sharma https://nadh.in/blog/open-source-is-not-broken/ https://nadh.in/blog/open-source-is-not-broken/ Sun, 12 Dec 2021 00:00:00 GMT I read this article (“Open Source” is Broken by Xe) written in the aftermath of the unfortunate log4j2 fiasco. The author discusses a pertinent problem that has plagued the FOSS (Free and Open Source) world ever since large for-profit corporations started their widespread consumption of FOSS, ever since countless “unicorns” raised infinite amounts of funding on valuations built pretty much entirely on FOSS, ever since FOSS got co-opted into corporatisation and capitalisation. And yet, countless maintainers of critical and widely used FOSS struggle to make a living.

]]> Kailash Nadh https://shrirangkahale.com/posts/proxmox-discord/ https://shrirangkahale.com/posts/proxmox-discord/ Fri, 10 Dec 2021 09:42:24 GMT Shrirang Kahale https://shrirangkahale.com/posts/dns-docker/ https://shrirangkahale.com/posts/dns-docker/ Fri, 10 Dec 2021 09:42:24 GMT Shrirang Kahale https://mrkaran.dev/posts/breaking-software/ https://mrkaran.dev/posts/breaking-software/ Thu, 09 Dec 2021 02:40:55 GMT followers = api.followers_ids(SCREEN_NAME) AttributeError: 'API' object has no attribute 'followers_ids' So, some method name changed. But that’s not all. The whole auth process including the initialisation of the API object changed as well. I’d spent some ~15 minutes grokking the docs but frustrated because 1) I don’t give a shit about v2/v1 APIs. 2) I just want to carry on with whatever I was doing. Why is this shit taking more time than I can care to give this to? I’d have cared enough if it was a side-project I maintained or something that I used daily. A utility like this which gets used once in a couple of years, will see more such breaking changes in future. Why, then should I spend migrating to v2 APIs, when after 2 years, v3 APIs would have broken my code again? What is the damn point? Why can’t software just keep working without troubling their users? In the end, I just installed the last version that works with v1 APIs and ran the script. I get breaking changes, I totally do. And I’ve no qualms with Tweepy. They did what they had to, in order to be compatible with v2. I am just angry/sad at the whole ecosystem of “Move fast, break things”. Please. Slow. Down. So that the rest of us who have a life can enjoy it and not spend an entire weekend migrating across versions! Sigh]]> Recently, I’d posted on Twitter that my feed has become messy overtime. Maybe I followed some accounts that were of no interest, maybe Twitter’s algos don’t really know what to show to me. Whatever, I wanted a fresh start. And, I’ve done this in past. I’ve removed all the accounts that I follow but they don’t follow me back. This little trick is helpful to start afresh while still not offending your friends ;)

I’d written a really simple Python script to do the job! It looks like:

import tweepy

SCREEN_NAME = 'mrkaran_'
CONSUMER_KEY = ''
CONSUMER_SECRET = ''
ACCESS_TOKEN = ''
ACCESS_TOKEN_SECRET = ''

auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
api = tweepy.API(auth)

followers = api.followers_ids(SCREEN_NAME)
friends = api.friends_ids(SCREEN_NAME)

for f in friends:
    if f not in followers:
        print("Unfollow {0}?".format(api.get_user(f).screen_name))
        api.destroy_friendship(f)

I’d last ran this in 2019 or so. That is dinosaur ages in the world of software. All I wanted to do was run this goddamn script again, 2 years later. Seems to be too much of an ask? Apparently tweepy the library used here to interact with Twitter’s APIs had a major release with lots of breaking changes. They’ve internally migrated to start using v2 Twitter API. So, when I naively ran pip install tweepy, my code threw:

❯ python main.py    
Traceback (most recent call last):
  File "/home/karan/Code/Personal/twitter-unfollow/main.py", line 13, in <module>
    followers = api.followers_ids(SCREEN_NAME)
AttributeError: 'API' object has no attribute 'followers_ids'

So, some method name changed. But that’s not all. The whole auth process including the initialisation of the API object changed as well. I’d spent some ~15 minutes grokking the docs but frustrated because 1) I don’t give a shit about v2/v1 APIs. 2) I just want to carry on with whatever I was doing. Why is this shit taking more time than I can care to give this to?

I’d have cared enough if it was a side-project I maintained or something that I used daily. A utility like this which gets used once in a couple of years, will see more such breaking changes in future. Why, then should I spend migrating to v2 APIs, when after 2 years, v3 APIs would have broken my code again? What is the damn point? Why can’t software just keep working without troubling their users?

In the end, I just installed the last version that works with v1 APIs and ran the script.

I get breaking changes, I totally do. And I’ve no qualms with Tweepy. They did what they had to, in order to be compatible with v2. I am just angry/sad at the whole ecosystem of “Move fast, break things”.

Please. Slow. Down.

So that the rest of us who have a life can enjoy it and not spend an entire weekend migrating across versions!

Sigh

]]> Karan Sharma https://ravidwivedi.in/posts/choosing-distro/ https://ravidwivedi.in/posts/choosing-distro/ Tue, 07 Dec 2021 00:00:00 GMT If you don’t know what a distro is, I will explain that in a minute.

First, I would like to ask you a question

Which operating system do you use?

Do you use Microsoft Windows?

Do you use MacOS?

or do you use Ubuntu?

Why do you use whatever you use?

In this post, I will share the reasoning behind choosing the operating systems I use and why it matters.

What is a distro? There are many operating systems known as GNU/Linux (actually they are known erroneously as “Linux”, please read this article for an explanation of why we call it GNU/Linux and not Linux). The members of the family of GNU/Linux systems are called distros.

I don’t use Microsoft Windows and MacOS because they are nonfree/proprietary software. And the symptoms of them being nonfree software are that they are malware and mistreat their users in many ways.

I do not want to use any nonfree software. So, for example, Ubuntu has nonfree software in its repositories and the version of Linux, the kernel, included in Ubuntu contains firmware blobs. That is not an ideal distro I would like to run.

Right now, I am using PureOS because it does not have any nonfree software in its repositories, does not ship with any nonfree firmware, follows Free System Distribution Guidelines (GNU FSDG) and so it is a GNU/FSF endorsed distro. Also, PureOS is ideologically inclined towards Free Software and values user’s freedom and privacy.

Further, PureOS is maintained by Purism, a company which is very committed to freedom and privacy of users. Purism is developing hardware, like mobile phones and laptops, which can run exclusively on fully free software. I support their work and when I say I use PureOS to someone, they might want to search about it and so it will raise awareness about Purism and their work.

Another distribution I use and endorse is Debian GNU/Linux as it does not ship with any nonfree firmware, it does not have any nonfree packages in its main repositories. I know GNU does not endorse Debian, but I think Debian is a good freedom-respecting distro. Like PureOS, Debian is also ideologically inclined towards free software philosophy. Debian adheres to Debian Social Contract as well which is committed to free software. Plus Debian has an inclusive community which is very welcoming to all. The decision-making of Debian Community is democratic in nature.

Further, I occassionally use other distros endorsed by GNU because they are committed to Free SOftware philosophy and contain no nonfree/proprietary software by default.

Summarizing the above discussion, I choose distros if they match the following criteria:

1. It should respect my freedom and privacy;

2. It should be ideologically inclined to free software philosophy.

A distro being a free software is not enough. When I use PureOS, other people will learn about Free Software and Purism. This is not the case with, say, Fedora or Ubuntu as they are not committed to free software (shipping with nonfree firmware by default is one example of this), although the operating systems are themselves Free Software. I suggest you the same if you support free software.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/covid-vaccine-profit-over-people/ https://ravidwivedi.in/posts/covid-vaccine-profit-over-people/ Sat, 04 Dec 2021 00:00:00 GMT Look at the image below:

What does the above image illustrate?

The image demonstrates how unequal the Covid-19 vaccine distribution has been across the countries.

Quoting the official data on Covid vaccines,

54.9% of the world population has received at least one dose of a COVID-19 vaccine. Only 6.2% of people in low-income countries have received at least one dose.

(on 4th December 2021)

Why is it so?

Is there a scarcity of vaccine units, or is the vaccine being developed at a slow rate?

No!

In fact, the vaccine production is accelerating, but the wealthy countries are restricting the access intentionally. Also, developed countries have preordered more number of vaccines than they require and there are only a few companies developing these vaccines, which means that people in low-income developing countries may not receive vaccinations from these manufacturers until 2023 or 2024.

Why this is so? This is where the profit of Pharmaceutical companies comes in and Bill Gates too.

Basically, the covid vaccine is patented (Patent is a legal right given to someone which excludes others from making or selling an “invention” ) and pharma companies other than companies who hold the patent on the vaccine cannot produce the vaccine.

When Oxford planned to allow all companies to manufacture its Covid vaccine, Bill Gates convinced them to cancel this plan. Another article. This is because Bill Gates prioritizes his profit over people’s health.

Next, Bill Gates came up with a scheme named Covax which, instead of removing the artificial restrictions from the vaccine and allowing independent companies to develop the vaccine, plans to charge high amount of money to wealthy countries and then donate the vaccine using to poor countries.

Companies participating in Covax will be able to set their own prices, without any transparency and accountability. They won’t have to face any legal liability for any potential damages caused by the vaccine made by them. This is what companies want– to earn as much as possible without any transparency and accountability of their actions.

So, basically, what is going on is that the taxpayers of rich countries are paying for higher prices set by pharma companies so that they can donate the vaccines to low-income countries rather than allow local manufacturers in poor countries to develop their own vaccine. The vaccine distribution could have been better in low-income countries if the local manufacturers were allowed to develop the vaccine. Poor countries are doomed to suffer because of the greed of corporates. Here is a good article explaining how patents are a barrier to vaccine distribution. Additionally, this scheme would also affect rich countries’ economies.

In fact, executive of one vaccine manufactuer said there would be a chance for them to raise prices for the vaccine when COVID moves from a pandemic state to an endemic situation and the virus circulates continually in pockets around the globe. When everyone is safe, the Covid will be over. Instead the company wants the opposite, so that the virus spreads and they can earn more profit.

The scenario is not special to Covid vaccine. Billionaires like Bill Gates have a large influence on public health policies. The root of the problem is capitalism– profit over people or in other words, the system which incentivizes greed over everything else. It is a customary practice for rich to invent schemes like this to remain rich at the expense on others. No surprise that they wouldn’t treat Covid vaccine any different.

Articles for further reading:

• How Bill Gates Impeded Global Access to Covid Vaccines

• How Pfizer Silences World Governments in Vaccine Negotiations

  The article mentions:

  • Pfizer silences governments through the use of nondisclosure provisions in many of its contracts. Brazil, for example, is prohibited from making “any public announcement concerning the existence… or terms” of the contract or commenting on its relationship with Pfizer without Pfizer’s prior written consent.

  • Pfizer can disallow governments from accepting additional donations of the Pfizer vaccine.

  • Pfizer exempts itself from liability for patent infringements, shifting the financial risk of Pfizer’s actions to government purchasers – despite Pfizer’s opposition to similar exemptions for manufacturers proposed at the World Trade Organization.

  • It gives the power to secret private arbitrators, not public courts, to decide issues on contract disputes.

  • Pfizer requires some countries to waive sovereign immunity, so it can go after state assets in case of a dispute.

  • Pfizer gives itself sole power when it comes to making key decisions, including how vaccine deliveries will be prioritized if there is a supply shortage.

  Another article for more information.

• Pressuring low-income countries.

• Moderna not sharing its technology stopping the poorest nations to produce the shot themselves.

]]> Ravi Dwivedi https://shrirangkahale.com/posts/whoogle-1/ https://shrirangkahale.com/posts/whoogle-1/ Fri, 26 Nov 2021 09:42:24 GMT Shrirang Kahale https://nadh.in/blog/on-powered-by-ai-marketing/ https://nadh.in/blog/on-powered-by-ai-marketing/ Fri, 26 Nov 2021 00:00:00 GMT An email I had sent in response to a survey on the use of “AI / ML” and the “AI-first mindset” in our organisation and in the industry was shared on social media^{[1] [2]} sparking surprising amounts of interest. I did candidly state the simple fact that we haven’t come across any big problems that warrant any specific “AI / ML” solutions in our organisation yet (Zerodha - stock broker that offers online investment and trading platforms), and that the bulk of the “powered by AI” claims we have seen across industries and in the numerous startup pitches that we receive, have been cases of hollow marketing. There is also a general expectation (delusion) that any sufficiently large technology company should be using “AI / ML” somehow, somewhere, for some reason.

]]> Kailash Nadh https://ravidwivedi.in/posts/free-software-explained-simply/ https://ravidwivedi.in/posts/free-software-explained-simply/ Thu, 25 Nov 2021 00:00:00 GMT

Hello World

I am Ravi

What color is this text?

When I write these lines in a text editor, save this file as test.html and open test.html in a browser, it looks like the following image:

Hello World

I am Ravi

What color is this text?

Now, if I change the color from Red to Green in the heading of the code above, save the file and open in the browser again. The heading becomes green in color. See the below screenshot for this code: Notice how change in code changes the color of the heading in the screenshots. You can do all sort of changes in the code and make the text whatever you like. Try copying the code in a text editor and save it as filename.html, where you can replace ‘filename’ by any name of your choice, and make changes in the text or color to play around and see it for yourself. The conclusion here is: The person/entity having source code of the software controls the software. For example, Microsoft has the source code of Microsoft Windows and therefore the software is in the control of Microsoft. The source code of Whatspp is with Facebook, so Facebook controls it. Users do not control the software if they do not have the source code. Defining Free Software With the above discussion, I wanted to illustrate that you cannot repair a software if you do not have the source code. You do not control the software if you do not have the source code. So, the user must have the source code so that they can inspect and modify it according to their needs. In the above example, if the source code were available with you, then you also require the freedom to share the software with people who know programming. It is analogous to you showing the refrigerator to a mechanic in the above example so that they can repair it. Now let’s say you have the source code of a software and the freedom to share it with others. There is a problem in that program you would like to fix. Someone who knows how to fix the problem, will modify the source code and give you their modified version so that your problem gets solved. When the software guarantees all the above-mentioned freedoms and, in addition, the freedom to run the software for any purpose, we call it Free Software. Free Software is a matter of liberty, not price. So ‘Free’ in ‘Free Software’ means freedom and not price. It is also called Libre Software or Swatantra Software or Mukt Software to emphasize that it is about freedom and not price. Think of free as in free speech not as in free meal. The following poster gives the definition of Free Software: Credits: Jeison Yehuda Amihud Source: https://www.gnu.org/graphics/amihud-4-freedoms.html LICENSE: CC-BY-SA 4.0 Examples of Free Software Check my Free Software List for some examples of Free Software. An example is Scribus, which is a desktop publishing software used in publishing newspapers, magazines etc. Scribus only supported publishing in Latin languages. Oman government funded adding a feature to Scribus, due to which Arabic support was added. Due to this, Scribus got Malayalam support and many newspapers now use Scribus to publish. The full story is here. It goes to show the value of freedom. Scribus is free software, which means it respects all the four freedoms mentioned in the poster. Source code being available, developers could add a feature and then used freedom 3 to share the modifications with others so that everyone, including nonprogrammers, benefitted from this. Another example is a free software named GNUKhata. Its development is funded by the Kerala government. Due to the freedom given by Free Software, GNUKhata is customizable. Due to its customizability, it now supports many Indian languages and many features like GST-compliance were added. Since many proprietary software for accounting are very costly and cannot be bought but can only be used against a license fee paid annually or periodically, many small businesses cannot afford them. GNUKhata can be downloaded for free-of-cost and users, businesses do not have to pay any license fee or ask for permission to use it. Think of what the world would have been without Free Software. What would have happened if there was no Free Software in the above mentioned examples. Credits: Thanks for Ravish, Praveen and Snehal for reviewing the article and giving suggestions.]]> The article was updated on 03 December 2021 to illustrate what source code means.

Let’s say your refrigerator has some problem, and it stopped working. You don’t know how to repair it. There is a repair shop in your town. The components of the refrigerator are sealed and there are no components available in the market. Every company makes their own components which are not compatible with other company’s refrigerators. That means you need to go to the same company to get it repaired, even though the mechanic in the shop near your house could have repaired it if it was not intentionally locked by the manufacturer. Unfortunately, it is weekend and the company services are not available on weekends.

Then you book an appointment on Monday and wait for a few days for your turn. When your turn comes, the company takes the refrigerator and “diagnoses” it for a few days. Then it returns you a call saying that it will take ₹ 50,000 to repair it, and you can get a brand-new refrigerator for ₹ 60,000. So, you got frustrated and buy a new refrigerator.

The story tells us the importance of freedom that we enjoy in our daily life. In the above story, if you had the freedom to see what’s inside your device. But it didn’t happen just because of the company making it intentionally locked and making sure that only they can repair all the products they sell.

Think of the same scenario in software. Let’s say you found a problem in the software you are working on. You are not a programmer, but other people who know programming can fix that problem. Every software is made of some human-readable code, which we call source code, which is then compiled by the computer. A software receives instructions from the code, and it does whatever the source code commands it to. You realized that the code on which the software is working is not available to you by the developer of that software. This leaves other programmers, except for the developer, in no position to fix the problem or even inspect what is wrong with the software. Similar to the refrigerator story above, the software source code is intentionally kept a secret from you and due to that restriction, you cannot get the problem fixed.

You try to contact the developer and report the problem, but there is no response. Now you try to switch to some other software, but then those software lacks some other crucial features which the previous one had. Now you are thinking that if you had the code, then there was a possibility to fix that problem. In this situation, you have to wait for the developer to fix the problem or use some other software or make your own and that too from scratch just to fix a problem, which would have surely been less hassle than to write the software from scratch.

Also, as we discussed earlier, a software takes commands from the source code. That means, whoever has the source code, has the full control over that software. If the user does not have the source code, then the user does not control the software.

What is source code?

Well, what is source code? To illustrate, I would like to show you a simple code. Don’t worry if you do not understand the code. I am only showing it to make my point.

See the following HTML code:

	<!DOCTYPE html>

	<html>

	<body>

	<h1 style="color:Red;">Hello World</h1>

	<p style="color:Blue;">I am Ravi</p>

	<p style="color:Purple;">What color is this text?</p>

	</body>

	</html>

When I write these lines in a text editor, save this file as test.html and open test.html in a browser, it looks like the following image:

	<!DOCTYPE html>

	<html>

	<body>

	<h1 style="color:Green;">Hello World</h1>

	<p style="color:Blue;">I am Ravi</p>

	<p style="color:Purple;">What color is this text?</p>

	</body>

	</html>

Now, if I change the color from Red to Green in the heading of the code above, save the file and open in the browser again. The heading becomes green in color.

See the below screenshot for this code:

Notice how change in code changes the color of the heading in the screenshots. You can do all sort of changes in the code and make the text whatever you like. Try copying the code in a text editor and save it as filename.html, where you can replace ‘filename’ by any name of your choice, and make changes in the text or color to play around and see it for yourself.

The conclusion here is: The person/entity having source code of the software controls the software. For example, Microsoft has the source code of Microsoft Windows and therefore the software is in the control of Microsoft. The source code of Whatspp is with Facebook, so Facebook controls it. Users do not control the software if they do not have the source code.

Defining Free Software

With the above discussion, I wanted to illustrate that you cannot repair a software if you do not have the source code. You do not control the software if you do not have the source code. So, the user must have the source code so that they can inspect and modify it according to their needs. In the above example, if the source code were available with you, then you also require the freedom to share the software with people who know programming. It is analogous to you showing the refrigerator to a mechanic in the above example so that they can repair it.

Now let’s say you have the source code of a software and the freedom to share it with others. There is a problem in that program you would like to fix. Someone who knows how to fix the problem, will modify the source code and give you their modified version so that your problem gets solved.

When the software guarantees all the above-mentioned freedoms and, in addition, the freedom to run the software for any purpose, we call it Free Software. Free Software is a matter of liberty, not price. So ‘Free’ in ‘Free Software’ means freedom and not price. It is also called Libre Software or Swatantra Software or Mukt Software to emphasize that it is about freedom and not price. Think of free as in free speech not as in free meal.

The following poster gives the definition of Free Software:

Examples of Free Software

Check my Free Software List for some examples of Free Software.

An example is Scribus, which is a desktop publishing software used in publishing newspapers, magazines etc. Scribus only supported publishing in Latin languages. Oman government funded adding a feature to Scribus, due to which Arabic support was added. Due to this, Scribus got Malayalam support and many newspapers now use Scribus to publish. The full story is here. It goes to show the value of freedom. Scribus is free software, which means it respects all the four freedoms mentioned in the poster. Source code being available, developers could add a feature and then used freedom 3 to share the modifications with others so that everyone, including nonprogrammers, benefitted from this.

Another example is a free software named GNUKhata. Its development is funded by the Kerala government. Due to the freedom given by Free Software, GNUKhata is customizable. Due to its customizability, it now supports many Indian languages and many features like GST-compliance were added. Since many proprietary software for accounting are very costly and cannot be bought but can only be used against a license fee paid annually or periodically, many small businesses cannot afford them. GNUKhata can be downloaded for free-of-cost and users, businesses do not have to pay any license fee or ask for permission to use it.

Think of what the world would have been without Free Software. What would have happened if there was no Free Software in the above mentioned examples.

Updated on Sunday 28 November 2021 to add suggestions by Praveen and Snehal.

Credits: Thanks for Ravish, Praveen and Snehal for reviewing the article and giving suggestions.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/what-does-facebook-outage-teach-us/ https://ravidwivedi.in/posts/what-does-facebook-outage-teach-us/ Thu, 25 Nov 2021 00:00:00 GMT On the 4th of October 2021, Facebook, Instagram, WhatsApp were down for more than 6 hours. This had a huge impact on many of its users, with devastating effect to many. During the outage, many users flocked to Twitter, Discord, Signal, and Telegram, resulting in disruptions on these apps’ servers.

Many businesses, which rely on Facebook’s services for their business, were down. In many countries, Facebook is synonymous with the internet and therefore their communications, business, payments, humanitarian work got disrupted. The outage temporarily broke the ability for some Facebook employees to access company buildings and conference rooms with their badges. And every third-party site that relies on “log in with Facebook” didn’t work as well.

What does this teach us? That Facebook’s services are centralized (they are controlled by a single entity). They are the single point of failure and control. A single company like Facebook controlling a large part of communications is no different from a dictator. You are at a mercy of them.

This tells us that the whole world relying on a single company for all their communications, businesses etc. is not sustainable.

In addition, such a service can be sold to any other company, like WhatsApp was sold to Facebook. So, even if the service is good today, it can become bad in the future– like, by selling it to another company or by changing privacy policy. Also, it is easier for governments to ban such a service.

Facebook is so bad for many reasons that my advice to you is to delete your Facebook account.

Now, what can we do about it?

I do not use or be user by any centralized service, be it Signal, Twitter, Telegram. I use decentralized and federated services. Email is an example of federation. A @gmail.com user can write mails to @yahoo.com user and vice-versa. This gives a choice of service providers, and therefore the entire communication systems of the world do not depend entirely on one service. If one service goes down, then this won’t disrupt the communications of the whole world.

Examples of decentralized networks are : Jitsi Meet(a video-calling software), Searx search engine etc. They can be self-hosted by anyone in their server. This gives users a choice of service providers.

Examples of federated networks are: Fediverse, XMPP, Matrix etc.

XMPP is a chat protocol. There are many services which a user can register with and still talk to another XMPP user who is registered on another XMPP service. This choice of service providers ensures that users are not locked into a single provider.

To get started with XMPP, you can use Quicksy on Android, whose onboarding process is similar to WhatsApp and Telegram. However, Quicksy users can talk to users registered on other XMPP providers, and so they are not locked-in to the service. If one service on XMPP shuts down or changes their policy, users can switch to other provider without going to the trouble of convincing every contact to switch to a new service. Compare this with WhatsApp, Signal, Telegram. If they change their policy or you disagree with these providers in future, you need to convince every contact to switch to that new service.

The bottom line is that Facebook’s outage is a reminder that they are a single point of failure, and other centralized services are no different in this aspect. Switch to decentralized and federated services instead, rather than being locked into a single provider for all your needs.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/scribus/ https://ravidwivedi.in/posts/scribus/ Wed, 24 Nov 2021 00:00:00 GMT Janayugom is a newspaper in Kerala, India, which publishes news in Malayalam language. Earlier they were using Adobe Pagemaker for publishing which supported only ASCII encoding. They considered using a better software for publishing in Malayalam. Many suggested them to try Adobe InDesign, but they realized that they need to pay a hefty subscription fee, which they could not afford. They came across community members of FSCI (Free Software Community of India), who suggested them to use Scribus for their publishing work. They don’t have to pay for any subscription fee to use it. Plus it supports Malayalam.

The developers of Scribus only added support for Latin languages, like English, Spanish, etc. Oman government funded adding support for Non-Latin languages in Scribus with Complex Text Layout feature, because they wanted support for Arabic language. This made is possible to add Malayalam support to Scribus.

This was done independent of the developers of Scribus. It is because Scribus is Free Software (free as in freedom, not price), which means anyone can adapt Scribus according to their needs. Such a thing is not possible with those Adobe software as they are proprietary and only developers can make changes.

What will you do if the developer of a proprietary software does not care about the feature you want to add? Maybe the developers don’t care about supporting your language. In the case of Free Software, such changes are possible, independent of developers’ wish. Further, the funded project by the Oman government helped people in India to publish in their language. It is because users get freedom to share the modified versions of Free Software. And thus, any addition of feature helps the whole society.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/the-importance-of-communities/ https://ravidwivedi.in/posts/the-importance-of-communities/ Sun, 21 Nov 2021 00:00:00 GMT Wikipedia defines ‘Community’ as:

A community is a social unit (a group of living things) with commonality such as norms, religion, values, customs, or identity.

Communities can be of many types and they might exist for different set of goals. In this post, I will share my experience with the free software commmunity that I am part of and its importance in my life. The topic has been touched in this blog post by FSCI earlier. The point was that communities can self-host and maintain free software powered services for the benefit of all because every individual self-hosting their services is not possible(for, say, people who do not know how to do) and not sustainable. Therefore, people can collaborate and run services.

That aspect is very important and that helps me greatly in my use of free software. In this post, I will touch on another aspect of the importance of communities, that is– the psychological boost that we get by being in a group which shares our goals. I continually say that I use only free software for my computing. But how is that possible in the era where proprietary software dominates? First comes my realization that software must respect user’s freedom. Then, my willpower to use only free/swatantra software. Then I also rely on the aspect touched earlier in the post about community-run services. I use community-run services(like the ones by FSCI) for my daily use. Abhas’ services helped me largely too. But one of the most important aspects is: that I am a part of the community which has a goal of promoting freedom-respecting software and full of people who understand the value of privacy. I think that if I was doing it all alone, I would probably would have been demotivated. I don’t know if I were able to keep it up. I keep getting demotivated but when I see that I am not alone in this activism, it definitely helps. Then I have friends from Free Software Communities outside of FSCI too. Many of them I know from Mastodon.

Also, I like how our community is inclusive and welcoming to all. Plus, it is nonhierarchial too. This aspect has made me hate hierarhical and centralized social structures as well(more on this later ;) ). We are also working on increasing the diversity in the community too. I hope that that people from all backgrounds (and not only privileged) get benefit and freedom that freedom-respecting software provides to the user.

I wanna thank you all for making this possible :)

Similar read: Arun Mathai also has a post on why he likes communities.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/upload-on-wikimedia/ https://ravidwivedi.in/posts/upload-on-wikimedia/ Wed, 17 Nov 2021 00:00:00 GMT If you have an image or a photo that you have taken which is useful for informative purposes, you can upload them to Wikimedia Commons.

All users of files found on Wikimedia Commons must be given the Four Freedoms:

• The freedom to use the media.
• The freedom to study the media and use information gained from it.
• The freedom to make and distribute copies of the media.
• The freedom to make changes to the media and distribute derived versions.

I publish my images under a copyleft license, which means if you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original, thus protecting every user’s (whoever uses that image) freedom. One such license is CC-BY-SA 4.0.

You can upload your images to Wikimedia Commons as well. Whenever you have some photo you shot and want to contribute it to public so that others can use it in their work, upload it to Wikimedia Commons.

A federated and decentralized version of image sharing Wikimedia Commons would be good. Currently it is centralized and all the images are on one server. We can store these images at many places to avoid single point of failure.

Thanks to sahilister, who suggested me to upload there.

]]> Ravi Dwivedi https://mrkaran.dev/posts/debugging-packer-ci/ https://mrkaran.dev/posts/debugging-packer-ci/ Mon, 15 Nov 2021 18:30:00 GMT {"changed": false, "msg": "groupadd: Permission denied.\ngroupadd: cannot lock /etc/group; try again later.\n", "name": "consul"} Erhm, okayyy. That looks like a permission error. But why does this not happen in my local was the question eating me up. Now was the time to start from the ground up and dissect different things going here. I will add a small hint though: The Gitlab CI runner is a Docker-based runner. That means all the commands like packer build etc happen inside a Docker container. I am using hashicorp/packer:light image, which is an Alpine based image containing just the packer executable. I tried to run the container locally with: docker run -v `pwd`:/app --rm -it --entrypoint='' hashicorp/packer:light sh And yes. When I ran packer build, I could replicate the issue here! But wait. More questions than answers. Ansible would run this groupadd command on the remote host, right? Why does Ansible care if it’s inside a container or not? So, I created a really simple playbook to reproduce this further. - name: Assemble Consul cluster hosts: localhost any_errors_fatal: true become: true become_user: root tasks: - name: Add Consul group group: name: debug_consul state: present I ran this ansible-playbook test.yml inside the container and… it worked! Okay, now it’s becoming clear. We aren’t executing ansible-playbook directly, it’s being wrapped by Packer. So this is clearly getting messed up by Packer. This time I did find a GitHub issue where people were asking about similar issues and this person described it well: I opened Packer docs again and that’s when I read this: We recommend against running Packer as root; if you do then you won’t be able to successfully run your Ansible playbook as root; become: yes will fail. WTF!!! This was right there in the docs, hiding in plain sight. Sheesh! Okay, so I can not run my playbook with become: true with the packer Image (which uses root user). Time to fix that by building a custom image. That is because Gitlab CI won’t let me change the user without hacking stuff. And a custom image also allows me to ditch Alpine for Ubuntu, which is what I prefer. FROM ubuntu:latest RUN apt-get update && apt-get install -y \ python3 \ git \ curl \ unzip \ python3-pip \ && rm -rf /var/lib/apt/lists/* RUN curl -o packer.zip https://releases.hashicorp.com/packer/1.7.8/packer_1.7.8_linux_amd64.zip RUN unzip packer.zip RUN mv packer /usr/local/bin RUN pip3 install ansible RUN useradd -rm -d /home/ubuntu -s /bin/bash -g root -G sudo -u 1000 ubuntu USER ubuntu WORKDIR /tmp ENV PATH="$HOME/.local/bin:$PATH" WORKDIR /app Using this custom image, the packer build worked just fine. Fun day indeed (/s). Fin!]]> Today I faced an issue that questioned my sanity. Since I didn’t find many “related” issues on StackOverflow/Google-fu except a lone GitHub thread where a kind stranger hinted at what could be the issue, I am writing about it here in the hopes (although I don’t wish this torture on anyone) that it helps someone!

To give some context, I am running a Gitlab CI job that bakes an AMI using Packer. Packer can use different kinds of provisioners to do configure stuff on the host and then prepare the image. I am using Ansible provisioner to install and configure Consul. At this point you may think this post is sponsored by Hashicorp by the sheer mention of all their products, but I assure you that is not the case.

Anyway, so this role works locally but it fails on the damn Gitlab CI. Classic case of Works on my machine, Ops problem now. These kinds of issues although are particularly exciting for me because they give me a chance to dig down deeper in the internals and slowly peel apart layers to figure out where the “drift” between local and CI is happening.

Here’s the relevant Packer snippet (Oh and this week as I updated myself with new Packer releases, it’s now possible to write Packer config with HCL and not just JSON anymore! Yayie. Again a reminder: not a sponsored post).

build {
  sources = ["source.amazon-ebs.golden-ami"]
  provisioner "ansible" {
    playbook_file           = "${var.playbook_file}"
    extra_arguments         = ["--tags", "install", "-e", "ansible_python_interpreter=/usr/bin/python3"]
    ansible_env_vars        = ["ANSIBLE_LOCAL_TEMP=$HOME/.ansible/tmp", "ANSIBLE_REMOTE_TEMP=$HOME/.ansible/tmp"]
    galaxy_file             = "${var.galaxy_file}"
    inventory_file_template = "[consul_instances]\n{{ .HostAlias }} ansible_host={{ .Host }} ansible_user={{ .User }} ansible_port={{ .Port }}\n"
  }
}

Running packer build in CI results in a failure of a task defined in the Ansible playbook. The task simply creates a new group:

- name: Add Consul group
  group:
    name: "{{ consul_group }}"
    state: present
  when:
    - consul_manage_group | bool

group runs groupapp command behind the scenes and you should be in the list of sudoers to actually create new groups. Since I already have become: true and become_user: root in my playbook that requirement is fulfilled. Moreover, this task runs just fine in the local as I mentioned above. While running in CI, I see the following error:

amazon-ebs.golden-ami: TASK [consul : Add Consul group] ***********************************************
amazon-ebs.golden-ami: fatal: [default]: FAILED! => {"changed": false, "msg": "groupadd: Permission denied.\ngroupadd: cannot lock /etc/group; try again later.\n", "name": "consul"}

Erhm, okayyy. That looks like a permission error. But why does this not happen in my local was the question eating me up.

Now was the time to start from the ground up and dissect different things going here. I will add a small hint though: The Gitlab CI runner is a Docker-based runner. That means all the commands like packer build etc happen inside a Docker container. I am using hashicorp/packer:light image, which is an Alpine based image containing just the packer executable.

I tried to run the container locally with:

docker run -v `pwd`:/app --rm -it --entrypoint='' hashicorp/packer:light sh

And yes. When I ran packer build, I could replicate the issue here! But wait. More questions than answers. Ansible would run this groupadd command on the remote host, right? Why does Ansible care if it’s inside a container or not? So, I created a really simple playbook to reproduce this further.

- name: Assemble Consul cluster
  hosts: localhost
  any_errors_fatal: true
  become: true
  become_user: root
  tasks:
    - name: Add Consul group
      group:
        name: debug_consul
        state: present

I ran this ansible-playbook test.yml inside the container and… it worked! Okay, now it’s becoming clear. We aren’t executing ansible-playbook directly, it’s being wrapped by Packer. So this is clearly getting messed up by Packer. This time I did find a GitHub issue where people were asking about similar issues and this person described it well:

I opened Packer docs again and that’s when I read this:

We recommend against running Packer as root; if you do then you won’t be able to successfully run your Ansible playbook as root; become: yes will fail.

WTF!!! This was right there in the docs, hiding in plain sight. Sheesh!

Okay, so I can not run my playbook with become: true with the packer Image (which uses root user). Time to fix that by building a custom image. That is because Gitlab CI won’t let me change the user without hacking stuff. And a custom image also allows me to ditch Alpine for Ubuntu, which is what I prefer.

FROM ubuntu:latest

RUN apt-get update && apt-get install -y \
    python3 \
    git \
    curl \
    unzip \
    python3-pip \
    && rm -rf /var/lib/apt/lists/*

RUN curl -o packer.zip https://releases.hashicorp.com/packer/1.7.8/packer_1.7.8_linux_amd64.zip
RUN unzip packer.zip
RUN mv packer /usr/local/bin

RUN pip3 install ansible

RUN useradd -rm -d /home/ubuntu -s /bin/bash -g root -G sudo -u 1000 ubuntu
USER ubuntu

WORKDIR /tmp

ENV PATH="$HOME/.local/bin:$PATH"

WORKDIR /app

Using this custom image, the packer build worked just fine. Fun day indeed (/s).

Fin!

]]> Karan Sharma https://www.evalapply.org/posts/systems-scale-value/index.html https://www.evalapply.org/posts/systems-scale-value/index.html Sat, 13 Nov 2021 00:00:00 GMT Aditya Athalye systems scale complexity https://www.evalapply.org/posts/hello-world/index.html https://www.evalapply.org/posts/hello-world/index.html Wed, 10 Nov 2021 00:00:00 GMT Aditya Athalye hello_world howto whyto https://mrkaran.dev/posts/load-testing-k6/ https://mrkaran.dev/posts/load-testing-k6/ Thu, 28 Oct 2021 18:30:00 GMT r.status == 200, }) || errorRate.add(1); sleep(0.5); } To run it: k6 run -d 10s -u 10 httpbin_load.js (Here -d is for the duration to run the test and -u is to specify Virtual User) Explanation: It’s an HTTP POST request with some form data to https://httpbin.org/post We use the uuid function because the JS stdlib is great at providing basic helper methods (/s) We define a check for HTTP status code as 200. Later we’ll see how to add more real-world checks under heavy load. We have a sleep function to pause a little bit before each iteration. This is pretty important as leaving sleep is akin to a user pressing F5 on a browser non-stop and you’d probably not want your load tests to be that aggressive. Read docs for more info. Output: ❯ k6 run -d 30s -u 10 test.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\ \ | (‾) | / __________ \ |__| \__\ \_____/ .io execution: local script: test.js output: - scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop): * default: 10 looping VUs for 30s (gracefulStop: 30s) running (0m30.6s), 00/10 VUs, 387 complete and 0 interrupted iterations default ✓ [======================================] 10 VUs 30s ✓ status is 200 checks.........................: 100.00% ✓ 387 ✗ 0 data_received..................: 318 kB 10 kB/s data_sent......................: 78 kB 2.5 kB/s http_req_blocked...............: avg=28.65ms min=210ns med=857ns max=1.1s p(90)=1.49µs p(95)=1.68µs http_req_connecting............: avg=7.77ms min=0s med=0s max=301.24ms p(90)=0s p(95)=0s http_req_duration..............: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms { expected_response:true }...: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms http_req_failed................: 0.00% ✓ 0 ✗ 387 http_req_receiving.............: avg=177.35µs min=39.43µs med=170.18µs max=604.22µs p(90)=259.75µs p(95)=285.73µs http_req_sending...............: avg=268.41µs min=48.62µs med=216.61µs max=7.95ms p(90)=334.94µs p(95)=448.21µs http_req_tls_handshaking.......: avg=15.87ms min=0s med=0s max=615.25ms p(90)=0s p(95)=0s http_req_waiting...............: avg=253.61ms min=215.34ms med=230.77ms max=826.57ms p(90)=316.98ms p(95)=324.69ms http_reqs......................: 387 12.666038/s iteration_duration.............: avg=784ms min=717ms med=732.43ms max=1.92s p(90)=820.35ms p(95)=926.87ms iterations.....................: 387 12.666038/s vus............................: 10 min=10 max=10 vus_max........................: 10 min=10 max=10 Things to look for: From the above output, I think these 2 metrics are the most important to look at: http_req_duration..............: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms http_reqs......................: 387 12.666038/s We see the total requests sent in 30s were 387 and the p95 response time is 325.1ms. Testing some real-world scenarios# This was a really simple example but we can add some more scenarios to mimic real-world checks. Let’s tweak the script to Go from 1 to 10 users in 10s. Stay at 10 users for 5s. Ramp down to 1 user for the next 15s. Have a threshold of not exceeding 500ms as p95. Have a threshold for the count of non 200 OK responses. The above script now becomes: import http from 'k6/http'; import { check, sleep } from 'k6'; import { Rate } from 'k6/metrics'; import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.0.0/index.js"; export const errorRate = new Rate('non_200_requests'); export let options = { stages: [ // Ramp-up from 1 to 10 VUs in 10s. { duration: "10s", target: 10 }, // Stay at rest on 10 VUs for 5s. { duration: "5s", target: 10 }, // Linearly ramp down from 10 to 0 VUs over the last 15s. { duration: "15s", target: 0 } ], thresholds: { // We want the 95th percentile of all HTTP request durations to be less than 500ms "http_req_duration": ["p(95)<500"], // Thresholds based on the custom metric `non_200_requests`. "non_200_requests": [ // Global failure rate should be less than 1%. "rate<0.01", // Abort the test early if it climbs over 5%. { threshold: "rate<=0.05", abortOnFail: true }, ], }, }; export default function () { const url = 'https://httpbin.org/post'; const params = { headers: { 'Content-Type': 'application/x-www-form-urlencoded', }, }; const data = { custname: "hello", comments: uuidv4(), }; check(http.post(url,data, params), { 'status is 200': (r) => r.status == 200, }) || errorRate.add(1); sleep(Math.random() * 1 + 1); // Random sleep between 1s and 2s. } Run with k6 run test.js: 2.21.0 on ☁️ (ap-south-1) took 24s ❯ k6 run test.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\ \ | (‾) | / __________ \ |__| \__\ \_____/ .io execution: local script: test.js output: - scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop): * default: Up to 10 looping VUs for 30s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s) running (0m30.3s), 00/10 VUs, 105 complete and 0 interrupted iterations default ✓ [======================================] 00/10 VUs 30s ✓ status is 200 checks.........................: 100.00% ✓ 105 ✗ 0 data_received..................: 126 kB 4.2 kB/s data_sent......................: 25 kB 841 B/s http_req_blocked...............: avg=73.69ms min=292ns med=761ns max=1.02s p(90)=1.54µs p(95)=688.76ms http_req_connecting............: avg=21.67ms min=0s med=0s max=245.46ms p(90)=0s p(95)=223.77ms ✓ http_req_duration..............: avg=252.72ms min=215.88ms med=230.31ms max=560.17ms p(90)=299.12ms p(95)=406.94ms { expected_response:true }...: avg=252.72ms min=215.88ms med=230.31ms max=560.17ms p(90)=299.12ms p(95)=406.94ms http_req_failed................: 0.00% ✓ 0 ✗ 105 http_req_receiving.............: avg=177.85µs min=103.82µs med=163.76µs max=366.5µs p(90)=235.86µs p(95)=266.39µs http_req_sending...............: avg=258.17µs min=96.92µs med=215.88µs max=958.67µs p(90)=410.05µs p(95)=487.18µs http_req_tls_handshaking.......: avg=50.22ms min=0s med=0s max=614.14ms p(90)=0s p(95)=460.3ms http_req_waiting...............: avg=252.29ms min=215.17ms med=229.69ms max=559.86ms p(90)=298.42ms p(95)=406.52ms http_reqs......................: 105 3.471037/s iteration_duration.............: avg=1.84s min=1.22s med=1.85s max=3.02s p(90)=2.22s p(95)=2.46s iterations.....................: 105 3.471037/s vus............................: 1 min=1 max=10 vus_max........................: 10 min=10 max=10 We can see that all the checks passed without breaching any thresholds we’d set. Some important points: In my local environment, I stress tested my service with 10k VUs which is quite a high number for the service but it was good to see it hold under extreme conditions as well. An important thing to note if you are spawning many VUs is that ulimit number should be high. This is described in their docs as well. To debug the HTTP response you can run with --http-debug="full" flag and get the verbose output for debugging. Summary# I’ve barely scratched the surface of what this tool does. You can export metrics to various data sources, add a lot more checks on the response code, use it with GRPC or WebSockets as well. Overall pretty happy with this tool and I am going to use more of it for future projects. References# https://k6.io/blog/comparing-best-open-source-load-testing-tools/ https://k6.io/our-beliefs/#simple-testing-is-better-than-no-testing https://k6.io/docs/ https://github.com/grafana/k6 Fin]]> This week I was occupied with optimising a Golang program I’d written at work. I wanted a way to reproduce the issue under heavy load on my development environment and Load Tests are a good way to do that.

The service in question is a RESTful API so it’s relatively easy to use any HTTP load test tools. The endpoint had an input parameter uuid which accepted a valid UUIDv4 as the input. To my surprise, this was not so straightforward with hey (which is my tool of choice for simple tests) and ab. While it was possible to write an external script to do that, I thought to look around at some “scriptable” alternatives. I found wrk which allowed me to write custom Lua modules. Now, I didn’t want to lose my focus from the main task which was load testing my service to write Lua, so I didn’t use wrk but it’s still a pretty decent option (and very very fast, at that).

Hello k6!#

Some more Google-fu resulted in me finding k6. I’d never heard of this but after exploring the GitHub repo and the docs it looks like a pretty active project.

So, k6 basically allows you to write scriptable tests which allow you to test a variety of scenarios. The scripts are written in Javascript and treated as ES6 Modules for extensibility. k6 has a concept of Virtual Users to mimic a real-world user. Each VU runs the “script” in an isolated self-contained JS runtime using Goja. Now obviously at this point, if speed is your utmost concern to generate very heavy load tests, I guess wrk is your only real choice as invoking a JS runtime inside Go won’t be super fast. But for most use-cases and people, like my case, this will just be fine.

Basic Usage#

Anyway, I quickly grokked the docs and copy-pasted some examples and modified them to what I needed. I was able to run a basic load test running very quickly and admired the simplicity here. It generated some p90, p95 etc stats which were helpful to look at. Here’s a basic example of how the script looks:

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate } from 'k6/metrics';
import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.0.0/index.js";

export const errorRate = new Rate('errors');

export default function () {
  const url = 'https://httpbin.org/post';
  const params = {
    headers: {
      'Content-Type': 'application/x-www-form-urlencoded',
    },
  };

  const data = {
    custname: "hello",
    comments: uuidv4(),
  };
  check(http.post(url,data, params), {
    'status is 200': (r) => r.status == 200,
  }) || errorRate.add(1);

  sleep(0.5);
}

To run it:

k6 run -d 10s -u 10 httpbin_load.js

(Here -d is for the duration to run the test and -u is to specify Virtual User)

Explanation:

• It’s an HTTP POST request with some form data to https://httpbin.org/post
• We use the uuid function because the JS stdlib is great at providing basic helper methods (/s)
• We define a check for HTTP status code as 200. Later we’ll see how to add more real-world checks under heavy load.
• We have a sleep function to pause a little bit before each iteration. This is pretty important as leaving sleep is akin to a user pressing F5 on a browser non-stop and you’d probably not want your load tests to be that aggressive. Read docs for more info.

Output:

❯ k6 run -d 30s -u 10 test.js                   

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: test.js
     output: -

  scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop):
           * default: 10 looping VUs for 30s (gracefulStop: 30s)


running (0m30.6s), 00/10 VUs, 387 complete and 0 interrupted iterations
default ✓ [======================================] 10 VUs  30s

     ✓ status is 200

     checks.........................: 100.00% ✓ 387       ✗ 0   
     data_received..................: 318 kB  10 kB/s
     data_sent......................: 78 kB   2.5 kB/s
     http_req_blocked...............: avg=28.65ms  min=210ns    med=857ns    max=1.1s     p(90)=1.49µs   p(95)=1.68µs  
     http_req_connecting............: avg=7.77ms   min=0s       med=0s       max=301.24ms p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms 
       { expected_response:true }...: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms 
     http_req_failed................: 0.00%   ✓ 0         ✗ 387 
     http_req_receiving.............: avg=177.35µs min=39.43µs  med=170.18µs max=604.22µs p(90)=259.75µs p(95)=285.73µs
     http_req_sending...............: avg=268.41µs min=48.62µs  med=216.61µs max=7.95ms   p(90)=334.94µs p(95)=448.21µs
     http_req_tls_handshaking.......: avg=15.87ms  min=0s       med=0s       max=615.25ms p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=253.61ms min=215.34ms med=230.77ms max=826.57ms p(90)=316.98ms p(95)=324.69ms
     http_reqs......................: 387     12.666038/s
     iteration_duration.............: avg=784ms    min=717ms    med=732.43ms max=1.92s    p(90)=820.35ms p(95)=926.87ms
     iterations.....................: 387     12.666038/s
     vus............................: 10      min=10      max=10
     vus_max........................: 10      min=10      max=10

Things to look for:

From the above output, I think these 2 metrics are the most important to look at:

     http_req_duration..............: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms 
     http_reqs......................: 387     12.666038/s

We see the total requests sent in 30s were 387 and the p95 response time is 325.1ms.

Testing some real-world scenarios#

This was a really simple example but we can add some more scenarios to mimic real-world checks. Let’s tweak the script to

• Go from 1 to 10 users in 10s.
• Stay at 10 users for 5s.
• Ramp down to 1 user for the next 15s.
• Have a threshold of not exceeding 500ms as p95.
• Have a threshold for the count of non 200 OK responses.

The above script now becomes:

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate } from 'k6/metrics';
import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.0.0/index.js";

export const errorRate = new Rate('non_200_requests');

export let options = {
    stages: [
        // Ramp-up from 1 to 10 VUs in 10s.
        { duration: "10s", target: 10 },

        // Stay at rest on 10 VUs for 5s.
        { duration: "5s", target: 10 },

        // Linearly ramp down from 10 to 0 VUs over the last 15s.
        { duration: "15s", target: 0 }
    ],
    thresholds: {
        // We want the 95th percentile of all HTTP request durations to be less than 500ms
        "http_req_duration": ["p(95)<500"],
        // Thresholds based on the custom metric `non_200_requests`.
        "non_200_requests": [
            // Global failure rate should be less than 1%.
            "rate<0.01",
            // Abort the test early if it climbs over 5%.
            { threshold: "rate<=0.05", abortOnFail: true },
        ],
    },
};

export default function () {
  const url = 'https://httpbin.org/post';
  const params = {
    headers: {
      'Content-Type': 'application/x-www-form-urlencoded',
    },
  };

  const data = {
    custname: "hello",
    comments: uuidv4(),
  };
  check(http.post(url,data, params), {
    'status is 200': (r) => r.status == 200,
  }) || errorRate.add(1);

  sleep(Math.random() * 1 + 1); // Random sleep between 1s and 2s.
}

Run with k6 run test.js:

2.21.0 on ☁️  (ap-south-1) took 24s 
❯ k6 run test.js

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: test.js
     output: -

  scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop):
           * default: Up to 10 looping VUs for 30s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)


running (0m30.3s), 00/10 VUs, 105 complete and 0 interrupted iterations
default ✓ [======================================] 00/10 VUs  30s

     ✓ status is 200

     checks.........................: 100.00% ✓ 105      ✗ 0   
     data_received..................: 126 kB  4.2 kB/s
     data_sent......................: 25 kB   841 B/s
     http_req_blocked...............: avg=73.69ms  min=292ns    med=761ns    max=1.02s    p(90)=1.54µs   p(95)=688.76ms
     http_req_connecting............: avg=21.67ms  min=0s       med=0s       max=245.46ms p(90)=0s       p(95)=223.77ms
   ✓ http_req_duration..............: avg=252.72ms min=215.88ms med=230.31ms max=560.17ms p(90)=299.12ms p(95)=406.94ms
       { expected_response:true }...: avg=252.72ms min=215.88ms med=230.31ms max=560.17ms p(90)=299.12ms p(95)=406.94ms
     http_req_failed................: 0.00%   ✓ 0        ✗ 105 
     http_req_receiving.............: avg=177.85µs min=103.82µs med=163.76µs max=366.5µs  p(90)=235.86µs p(95)=266.39µs
     http_req_sending...............: avg=258.17µs min=96.92µs  med=215.88µs max=958.67µs p(90)=410.05µs p(95)=487.18µs
     http_req_tls_handshaking.......: avg=50.22ms  min=0s       med=0s       max=614.14ms p(90)=0s       p(95)=460.3ms 
     http_req_waiting...............: avg=252.29ms min=215.17ms med=229.69ms max=559.86ms p(90)=298.42ms p(95)=406.52ms
     http_reqs......................: 105     3.471037/s
     iteration_duration.............: avg=1.84s    min=1.22s    med=1.85s    max=3.02s    p(90)=2.22s    p(95)=2.46s   
     iterations.....................: 105     3.471037/s
     vus............................: 1       min=1      max=10
     vus_max........................: 10      min=10     max=10

We can see that all the checks passed without breaching any thresholds we’d set.

Some important points:

• In my local environment, I stress tested my service with 10k VUs which is quite a high number for the service but it was good to see it hold under extreme conditions as well. An important thing to note if you are spawning many VUs is that ulimit number should be high. This is described in their docs as well.

• To debug the HTTP response you can run with --http-debug="full" flag and get the verbose output for debugging.

Summary#

I’ve barely scratched the surface of what this tool does. You can export metrics to various data sources, add a lot more checks on the response code, use it with GRPC or WebSockets as well.

Overall pretty happy with this tool and I am going to use more of it for future projects.

References#

• https://k6.io/blog/comparing-best-open-source-load-testing-tools/
• https://k6.io/our-beliefs/#simple-testing-is-better-than-no-testing
• https://k6.io/docs/
• https://github.com/grafana/k6

Fin

]]> Karan Sharma https://ravidwivedi.in/posts/free-software-important-for-privacy/ https://ravidwivedi.in/posts/free-software-important-for-privacy/ Tue, 26 Oct 2021 00:00:00 GMT Free Software means software which respects user’s freedom. It does not mean that the users get software without paying or for free of cost. Free Software is a matter of liberty not price. I sometimes call it swatantra/mukt software to clarify this point.

Precisely, Free Software gives users the following freedoms:

• Freedom 0: Freedom to run the software;

• Freedom 1: Freedom to study and modify the software. Users must have the source code of the software to exercise this freedom;

• Freedom 2: Freedom to share the software;

• Freedom 3: Freedom to share your modified versions;

A software which lacks any of these freedoms is called nonfree/proprietary software. Most of the well-known proprietary software are malware.

If we do not have the source code of the software, we cannot inspect it for whether it has malicious functionalities. In particular, we cannot inspect whether the software has spyware or not. Even if we get the source code, we cannot remove the spyware unless we can modify it. In view of this note, freedom 1 is a precondition to user privacy.

If users cannot share the software, then they need to inspect the software themselves to know whether software has some spying functionality or not. Sharing the software allows us to give a copy of software to someone who can inspect. For example, nonprogrammers can share a copy of the software with someone to get the software inspected for malicious functionalities, maybe, in exchange for a fee. When we don’t know how to repair a fan, bicycle or a car and we give it to a mechanic who repairs for us. Similar is the case for software.

Let’s say someone removed a spyware from a software, then they can share their modifications with others if Freedom 3 is granted. If Freedom 3 is not granted, then everyone need to modify themselves which is a lot of redundant work.

These freedoms give users collective control over the software. The collective control is necessary for users to get privacy. A nonfree software cannot be trusted for privacy. Please note that Free Software might not be sufficient for privacy. It is a precondition for privacy.

For example, Ubuntu operating system is a Free Software and it contained spyware in older versions. With Free Software, users have a defence to remove those malfunctionalities. With proprietary software, there is no such chance.

Another factor is decentralization. What I mean by decentralization is that network-based services allow self-hosting and federation. Please read this article by FSCI for details on how decentralization in combination with Free Software and end-to-end encryption can give you privacy.

I will summarize the article for you here. Basically, when we use software installed in someone else’s computers(called servers), like Google Docs, we lose control over our computing. This is because all communications, say, in case of Google Docs will go via Google’s servers. Since, we are using Google’s computers for our computing they can log our activities and put us in surveillance. If the server is in our control or a trusted one, then this is not a problem but services like Google Docs does not allow users to deploy it on their own server(even if it would it would not be advisable due to Google Docs being nonfree software). So we need the freedom to self-host, which means freedom to run our own servers. This does not mean we have to run our own server. Granting users the freedom to self-host gives rise to many service providers. Then we can choose one which we trust, or pay someone to deploy our server.

Federation means two users using different service providers can communicate. This is required, otherwise when we switch the service provider, we need to make effort to switch every contact to a new service provider. Federation allows us to switch service provider without other contacts switching the provider.

Example of free software powered federated systems are Matrix, XMPP, Mastodon. Searx search engine is an example where self-hosting is allowed but the concept of federation does not apply.

To illustrate, we take an example of Matrix chat system. A user registered on service provider 1 can contact with users on service provider 2 on Matrix. All messages are end-to-end encrypted. And since the servers are under user’s control, they can control the policies and what data is being collected.

The conclusion is: Free Software and decentralization are necessary for privacy. Proprietary Software and centralized services cannot be trusted for privacy.

]]> Ravi Dwivedi https://nadh.in/blog/javascript-ecosystem-software-development-are-a-hot-mess/ https://nadh.in/blog/javascript-ecosystem-software-development-are-a-hot-mess/ Sat, 16 Oct 2021 00:00:00 GMT I have a small Vue 2 project (an admin UI for dictmaker) that I created with vue cli six months ago. Today, I picked it up again to finish it, and started out by doing a yarn upgrade. Of course, blindly upgrading all dependencies is never a good idea, but this is a tiny WIP project with just one dependency that I added, and there is a constant stream of GitHub dependabot alerts every month forcing me to upgrade some dependency or another, so what is the worst that could happen? At least that is what I thought.

]]> Kailash Nadh https://ravidwivedi.in/posts/problems-with-protonmail/ https://ravidwivedi.in/posts/problems-with-protonmail/ Wed, 29 Sep 2021 20:36:19 GMT Protonmail claims to automatically send end-to-end encrypted messages between all the Protonmail users. They claim that they do not have access to user’s private keys (Someone having access to your private keys can decrypt and read all your encrypted messages), by encrypting the user’s private keys with the hash generated by user’s password (which is known only to the user). The problem with this is: how do we know that Protonmail does not keep a copy of user’s private keys with themselves before encrypting it with user’s password hash? Also, we cannot really inspect the web application we use in the browser because it is not installed in our own devices, we are using it from Protonmail’s computers.

There is no need to trust server for email encryption. Email encryption can be done within the app. This method encrypts mails within the app before sending it to the server. Here is a guide to encrypt mails using pEp app. pEp encrypts mails very easily. You just have to send one mail to a pEp user to exchange keys and after that, all emails will be encrypted by default. Email encryption used to be hard to use, but it is no longer the case. pEp project has made email encryption (it uses OpenPGP) easier so that nontechnical people also have access to email encryption.

This is the main problem with Protonmail. When it comes to encrypting emails, there is no inherent need for the user to trust the server side for encryption, yet their whole model of encryption lies in trusting the server side. We cannot inspect their server side. We cannot inspect what they run or how they implement all this in their own computers. You can, however, inspect the app running in your own device if the app is free software.

Conclusion: Email encryption does not require user to trust the service provider. Users can encrypt emails in their device before sending the mail. Protonmail is making their users trust the server side for email encryption without any need for that. I advice against using emails from such email providers and instead use email providers which allow you to use independent app like pEp, Thunderbird, K9 Mail, so that you can encrypt emails before sending it to the server.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/story-of-this-website/ https://ravidwivedi.in/posts/story-of-this-website/ Tue, 28 Sep 2021 00:00:00 GMT This is the story of creation of the website you are reading and how it was created.

I always wanted to create a website from as far as I can remember. This is because I wanted autnonomy over the content that I can post. Social media posting format is not suitable for website like this. Also the website has a cool domain on my name which feels really good.

In the Software Freedom Camp 2020, I met Sahil and Arun. Sahil already had a website, while Arun purchased a domain name around that time. I got inspired from Arun and bought this domain from gandi.net in October 2020.

Then I took the project in Free Software Camp 2020 to learn Hugo. Didn’t learn much from there as I didn’t put a lot of time and effort into it.

After the domain purchase, the website wasn’t up for 9 months. Finally, in July 2021, Sahil helped me set up this website. And after tinkering with the website, I learnt enough Hugo and git to maintain the website. And here I am, writing the story of the website.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/fscamp-2020-as-learner/ https://ravidwivedi.in/posts/fscamp-2020-as-learner/ Mon, 27 Sep 2021 00:00:00 GMT Index

• Introduction

• What is Free Software Camp

• Motivation to join the camp

• How I came to know about the camp

• Experience as a learner in the camp

• Aftermath

• Word for learners of the upcoming camps

Introduction

I participated as a learner in Free Software Camp 2020 organized by FSCI and FSF India. This is a post about my experience of the same. If you don’t know what free software is, then you can read my article on the same. I have never attended any software-related camp other than this one and therefore I have no idea what other camps are about and so I cannot compare this camp with any other camp of technical nature or one which involves introducing free software philosophy to the participants. Since the camp concluded a long before I am writing this post, there is a chance I forgot a lot of things about it or my feelings/opinions of many things at that time.

What is Free Software Camp

This is my takeaway for what free software camp means. I think that the camp had two goals:

• To acquaint the participants with free software philosophy.

• To connect the learners with mentors and contribute to free software projects. These contributions don’t have to be technical in nature but they must contribute to free software in some way.

The camp was divided into the following phases:

• Ice breaking sessions.

• Introduce the learners with free software philosophy.

• GNU/Linux Installation phase.

• A phase where we get to meet people who earn their living from free software.

• Projects phase.

Motivation to join the camp

I was already introduced to free software philosophy and started caring about the idea in 2020. The main thing I was looking at the time around September 2020 was: How does free software actually work in reality? I was not aware of communities that power free software. Curiosity to know how free software works was one motivation to join the camp.

Also, before the start of the camp, I already started shifting to free software but I needed help on: 1. Which software do people from free software community use? ; 2. To try and test these software.

For example, I came to know about BigBlueButton from the camp. It is a videoconferencing software which respects freedom and allows self-hosting.

Another motivation was to meet with like-minded people because usually people don’t get excited about free software and therefore, one can get demotivated.

To sum up, my motivations were:

• To understand how free software communities work.

• To know what software free software proponents use.

• To meet like-minded people.

My motive was not really learning anything technical.

How I came to know about the camp

The camp was announced in mid September 2020. And a few days ago, I filled the volunteer form for FSF India. Due to that group’s announcement, I came to know about the camp.

Experience as a learner in the camp

I registered for the camp and started joining the sessions when the camp commenced. In the first stage, the learners were divided into small groups based on their language preference. I was added to Group 6 which was a Hindi group and Sruthi was assigned to our group from the organizing team as a co-ordinator(Sorry I cannot find a word for this).

I mixed up well with this group. We used to have regular meetings and shared what we think about various issues. People used to feel autonomous in sharing their thoughts. And we also helped each other in various things, for example, I helped people in setting up their gpg keys and taught how to send encrypted emails. So, it can be said that Group 6 ice-breaking session really broke the ice or maybe the ice was not even there to begin with. Groups were also given an activity to present on a topic assigned by the organizers. Our group decided to have an audio presentation on the topic. The topic was something related to democracy and digital rights(how surveillance, censorship etc. affects democracy). We all rehearsed it many times, suggested each other of points and finally completed the recording and submitted it for the presentation in a camp session. I found that overall, be it my group, other learners, organizers, or mentors, people were very helpful, kind and inclusive. This ended our ice-breaking sessions.

Next was GNU/Linux installation phase. I used to have Macbook at that time. I could boot GNU/Linux in my Macbook but the touchpad didn’t work. So that was a bit sad. I badly wanted to run a GNU/Linux distro on my Macbook. Maybe external mouse would have worked. But I didn’t try that. Therefore, I proceeded in the camp with MacOS installed.

Next was the phase where we met people who earn their living from free software. I attended sessions by Abhas and Nagarjuna. They were good and gave me some insights. Abhas has a way of making sessions interactive which I would like to do in my own sessions(not only in camp but anywhere). Nagarjuna raised issues about copyright mainly and was very interactive, which I really liked. He was in no hurry but asked questions from us and patiently waited for our responses and cleared doubts.

Now came the projects phase. Initially, I was not excited to do any technical project. So I thought of choosing a non-technical project. I ended up choosing a technical one because I wanted to learn how to create website so that I can create my own wesbite. I chose the project offered by Karthik to learn Hugo Static Site Generator to create websites. The project was to create and maintain Privacy Yathra website. I took the project as a long-term work rather than just set up the website as a part of the camp and run away.

I didn’t work a lot on the project and I could not learn Hugo within the deadline set by the camp. But later on, I set up this website you are reading, with the help of Sahil. From there on, I learnt from different people about different things about creating and maintaing the website. The Privacy Yathra website(which was the project of the camp) is not yet created but as now I know how to work with Hugo and git(actually I still don’t know much), I would one day be up. Keep looking for Privacy yathra website, folks :) It can be up anytime.

Shout out to all the people who made this work so smoothly. Especially organizers were always available for any help I needed and were very welcoming.

Aftermath

After the conclusion of the camp, I am integrated with FSCI as a campaigner. As mentioned above, I set up my website after the camp. Also, I am a part of the organizing team of the succeeding camp named “Software Freedom Camp Diversity Edition 2021” organized by FSCI. This year(2021) we are looking for diversity. We are focusing on reaching out to groups underrepresented in the free software community. Registrations for learners are open till 15th October. Please register if you are interested and from an underprivileged backgroud. Please help us raise awareness about the camp so that we can reach the needy. You can share the camp poster on social media(even on Twitter, Facebook, Instagram, if you use them anyway, then why not?), with your contacts in chat, emails etc.

Word for learners of the upcoming camps

If you are participating as a learner in any of the editions of the camp, then I have some advice for you. Hopefully, the above text gave you some idea on what happens in the camp. I think the best way to contribute to the camp is to contribute to free software. You might be surprised that the best way to contribute to free software is not about adding code or features to free software, although they are very good contributions. I think it is not a good idea to improve the code of some free software or work on a free software project for a few months and then forget about free software. I would suggest you to understand what free software philosophy is, why free software is important, use free software in your own lives for your own work and the hardest part, try to convince others to use free software and avoid proprietary software. If you are committed to free software philosophy, you will develop free software and contribute to free software as a side effect. If you are not committed to free software philosophy, then you will only contribute code to free software as a part of the camp and then forget about it. Take it as a lifetime project to contribute to free software to switch to free software and writing code to develop or improve the free software. Also, avoid proprietary software.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/fs-and-oss-same/ https://ravidwivedi.in/posts/fs-and-oss-same/ Fri, 24 Sep 2021 22:10:00 GMT Have you ever wondered what the difference is between the terms “free software” and “open-source software?” Perhaps, you think they are the same. It is often misunderstood that “free” refers to free of cost. The term “FOSS” is also very common. It expands to Free and Open Source Software. In addition, I have heard people say that since the software is open source and available free of cost, therefore it is denoted by the term FOSS. In this post, I’ll illustrate the difference between the two terms.

Free Software refers to software which respects users freedom to run, study, modify, share and share the modified versions of the software. For example, VLC media player, Firefox browser, and Emacs. “Free” in Free Software refers to liberty, not price. The software I just mentioned also qualify as Open Source. The definition of Open Source can be read on the opensource.org website by visiting this link. Since it is a bit too long, I didn’t include it here.

Practically speaking, the terms “free software” and “open source software” refer to the same class of software. However, the main difference between the two terms lies not in the actual software, but the values behind the respective movements. Free Software movement campaigns for users’ freedom and that users must control the software running in their own devices. On the other hand, Open Source movement does not put the issue in ethical terms like users’ freedoms or rights, but more of like a practical choice.

The term “free software” existed since 1984, while the term “open source” came into existence in 1998. Let’s look at why this new term was introduced even though the term “free software” was already there.

Quoting Open Source Initiative on coining the term open source,

The “open source” label was created at a strategy session held on February 3rd, 1998 in Palo Alto, California, shortly after the announcement of the release of the Netscape source code. The strategy session grew from a realization that the attention around the Netscape announcement had created an opportunity to educate and advocate for the superiority of an open development process. The conferees believed the pragmatic, business-case grounds that had motivated Netscape to release their code illustrated a valuable way to engage with potential software users and developers, and convince them to create and improve source code by participating in an engaged community. The conferees also believed that it would be useful to have a single label that identified this approach and distinguished it from the philosophically- and politically-focused label “free software.” Brainstorming for this new label eventually converged on the term “open source”, originally suggested by Christine Peterson.

It is apparent from the above-mentioned quote that the term “Open Source” was coined as a way to market the same idea represented by the term “Free Software” in a way detached from philosophical or moral considerations. It was promoted as a superior model of software development, not as a matter of user rights or freedom.

My friend and fellow Free Software activist Praveen has put this difference into a nice quote, which is very relevant to distinguish the two:

Open Source wants to create better software, Free Software wants to create a better society.

Quoting Richard Stallman from his famous article on the difference and the dangers that the approach open source movement has in avoiding the ethics part:

When open source proponents talk about anything deeper than that, it is usually the idea of making a “gift” of source code to humanity. Presenting this as a special good deed, beyond what is morally required, presumes that distributing proprietary software without source code is morally legitimate.

This approach has proved effective, in its own terms. The rhetoric of open source has convinced many businesses and individuals to use, and even develop, free software, which has extended our community—but only at the superficial, practical level. The philosophy of open source, with its purely practical values, impedes understanding of the deeper ideas of free software; it brings many people into our community, but does not teach them to defend it. That is good, as far as it goes, but it is not enough to make freedom secure. Attracting users to free software takes them just part of the way to becoming defenders of their own freedom.

Sooner or later these users will be invited to switch back to proprietary software for some practical advantage. Countless companies seek to offer such temptation, some even offering copies gratis. Why would users decline? Only if they have learned to value the freedom free software gives them, to value freedom in and of itself rather than the technical and practical convenience of specific free software. To spread this idea, we have to talk about freedom. A certain amount of the “keep quiet” approach to business can be useful for the community, but it is dangerous if it becomes so common that the love of freedom comes to seem like an eccentricity.

The difference between the two movements is best illustrated by an incident in which TiVo - a video recorder - used a software which was free/open-source and gave users legal right to modify it, but the hardware won’t allow users to run their modified versions. This restriction was fine for Open Source proponents like Linus Torvalds, but Free Software proponents like Stallman opposed such restrictions. A new version of GNU General Public License was written by the Free Software Foundation to address this issue. For more details on this, I would suggest you to read this page on the GNU website. Moreover, you can read the Wikipedia page too by clicking here.

Personally, I use the term “Free Software” and not “Open Source” due to my opposition against proprietary software. Furthermore, I agree with the ethical values associated with the term “Free Software”. Using this term helps me spread awareness about the difference and propagate the ethical values.

Feel free to use the term you prefer, but it is important to understand what the term means.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/drm/ https://ravidwivedi.in/posts/drm/ Thu, 23 Sep 2021 00:00:00 GMT Ever used Spotify, Apple Music, Netflix or Hotstar? Ever wondered why you cannot copy the files from them and share it with someone? Ever wondered why you cannot download the files in your device and transfer in another device or play them in any other media player? Let’s be clear on this: Downloading means that the file is now in your device and you can play/view it in any app of your choice. Download does not mean that you can only view the file in that particular app(like Spotify or Netflix) for offline use.

The answer is DRM.

DRM is usually known as Digital Rights Management by publishers, record labels and streaming dis-services. The long form Digital Restrictions Management is closer to reality.

DRM basically means digital files(any form of artwork, PDF, music files or software) are encrypted in such a way that users cannot copy them. This threatens user’s freedom because technology allows us to copy files and share with others. What is the point of “technological progress”(being able to copy files is an example of technological progress) if you cannot even use it?

Copying is an inherent nature of digital files to the extent that it reminds of the quote by Bruce Schneier, “Trying to make digital files uncopyable is like trying to make water not wet.” Also, since we can copy files without any marginal cost and share it to everyone, it raises a fundamental moral question, quoting Eben Moglen, “If I can provide to everyone all goods of intellectual value or beauty, for the same price that I can provide the first copy of those works to anyone, why is it ever moral to exclude anyone from anything? If you could feed everyone on earth at the cost of baking one loaf and pressing a button, what would be the moral case for charging more for bread than some people could afford to pay?”

The usual logic in favor of DRM is that artists lose money when someone copies a file and share it with someone. This argument assumes that everyone who has a copy of the file would have paid for a copy if sharing was prohibited. In reality, a lot of files would not even have been sold. Imagine a friend shared a book with you and you read it. Would you have read all such books shared with you if sharing was prohibited? When someone refuses sharing a book or a music file with you, do you go to market and purchase that thing? Also, when the marginal cost of copying is zero, nobody is losing money when you copy the file. Compare this situation with physical objects like chairs, noodles, clothes etc. Every piece of that physical object has a production cost while for software and digital files,the cost is only in the development of that file or software but every copy is effectively free-of-cost(given that computers and mobile phones are household items that people use for personal use anyway, so the cost of device is not included in the copying).

Note that in the current system, only well-established artists really make money. Usually, the artists do not get fair compensation and the streaming companies are the one earning from artists’ work. These DRM imposing dis-services act like a middle agency between users and artists, effectively robbing user’s freedom and artists’ pay.

Should we prohibit users from copying files and sharing them OR should we change our business model and allow them to share the files? Since, DRM benefits only publishers, streaming services, record labels and the like, while others receives discrimation, exclusion and lack of freedom to use technology, I think the the business model needs to change rather than restrciting users from sharing.

DRM is also being used in educational resources now-a-days which excludes people who can’t afford them or people who value freedom and won’t use files containing DRM.

DRM is already being used to discriminate against the poor in many ways.

When a DRM imposing service gets shut down, users lose access to all their data they purchased from that service. For example,Yahoo music used to distribute music files with DRM and when it shut down, users could not backup their files and lost access to all the music they purchased. Another example is when Microsoft’s ebook store closed, all the users lost their ebooks purchased from there.

Should we wait for the companies to eliminate DRM from the files? No. Companies might never remove DRM from their services. Rather, we can refuse to use services which impose DRM. You can use youtube-dl to download music files from many sites. You can use Newpipe Android app to download music and videos from YouTube. For educational resources, we can use Libgen or Scihub to download the books and academic papers. Check the DRM-free guide by FSF for websites which provide DRM-free content.

Everytime you pay for a DRM product, the companies building such products get funds, either by your data or you pay them directly. The measure of refusing to use DRM services brings change to an extent– that we do not fund such practices. But the real change is to have a model for authors/creators to get funded and they release the content without DRM.

We can fund artists without DRM. Fans can directly fund artists for their work. A lot of YouTubers get money from their Patreons and people do directly fund creators.

Here are a few examples to demonstrate that the model can work:

• Louis C.K. released copies of his film without any restrictions or DRM and the film got profit in 12 hours.

• Diesel Sweeties released a DRM free webcomics and had huge success. They started releasing DRM-free payment optional model since then.

• The band Radiohead released an album on their website without any restrictions with listeners allowed to pay nothing or any amount they would like to.

Therefore, we can fund artists directly and copy creative works will freedom. It is possible to have both freedom and creative work. What do you think?

]]> Ravi Dwivedi https://ravidwivedi.in/posts/phantoms-in-the-brain/ https://ravidwivedi.in/posts/phantoms-in-the-brain/ Thu, 09 Sep 2021 00:00:00 GMT A few days ago, I finished reading the book Phantoms in the Brain by V. S. Ramachandran. The topic covered here is fascinating, interesting, mysterious, disturbing and shocking. The cliche “Reality is stranger than fiction” comes to mind.

The book covers many disorders and tries to explain them in concrete naeurological terms.

One of the cases presented in the book was that majority of amputees who suddenly lost a limb in an accident feel pain in the lost limb. Many patients feel that the fingers of the lost limb (fingers are not even attached to the body) are clenched tightly and this leads to unbearable pain (called phantom pain and hence the title of the book).

In the past, neurologists regarded such cases as mental problems and handed over the patient to psychiatrists. The book holds the view that due to damages in specific portions of the brain leads to loss of functionality which that portion carries. This is fascinating because when trying to understand these cases, it seems like we are only looking at exceptional cases, but at the same time, they do tell a lot about the normal human brain. Ramachandran brought a fresh appraoch and looked at how phantoms (the patients still feel that the lost limb is attached to the body and such a limb is called a phantom limb) are generated and how the brain can be tricked into unlearning a phantom. Would you like to look at the solution/treatment of the phantom pain? It is here. The solution is interesting as well.

Human brain can easily be tricked. Our brains carry so many cognitive biases, delusions, denial of the obvious facts. Quoting Oliver Sacks from the preface of the book, “The deeply strange business of mirror agnosia, and that of misattri­buting one’s own limbs to others, are often dismissed by physicians as irrational. But these problems are also considered carefully by Rama­chandran, who sees them not as groundless or crazy, but as emergency defense measures constructed by the unconscious to deal with sudden overwhelming bewilderments about one’s body and the space around it. They are, he feels, quite normal defense mechanisms ( denial, repression, projection, confabulation, and so on) such as Freud delineated as universal strategies of the unconscious when forced to accommodate the intolerable or unintelligible.”

The topic presented here aligns very well with Daniel Kahneman’s Thinking, Fast and Slow, which says that there are two modes of thinking – System 1 and System 2. System 1 is fast, effortless and jumps to conclusions. It cannot be consciously controlled by us but we can change our habits to feed them into System 1 (like driving, walking, playing a cricket ball are in System 1 of the person practicing enough). System 2 is slow, effort and resource consuming. Heavy cognitive tasks are done using System 2.

Overall, Phantoms in the brain urges neuroscientists to be more open in their approach, while at the same time being fun to read, engaging and accessible to the general audience.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/rooting-my-phone/ https://ravidwivedi.in/posts/rooting-my-phone/ Tue, 07 Sep 2021 00:00:00 GMT (Warning: Installing a custom ROM on your device will void your warranty and there is a risk that it can brick your device)

(Note: In my case, I could afford to risk bricking my phone)

Index

• What is rooting?

• My experience of rooting

• How much free software does iode OS have

• How to install a custom ROM

What is rooting?

A user having root access of an operating system means they have the capability to make unrestricted changes in the system. Such a user is also called a superuser or an admin. Check out the wiki page for more details.

Mobile phones which ship with Android pre-installed in their devices do not give users root access by default. For example, the users have read-only permissions for /system/ partition on the phone. Therefore, users need to put effort into taking root access of their devices. This extra effort is what we call ‘rooting’. Check this Wikipedia page for a good list of advantages that rooting provides to users. This gives users full control over their devices.

Android phones are shipped with nonfree software preinstalled which users cannot remove without having root access. Apps like YouTube, Google Play Store and other malware by Google are usually preinstalled. I wanted to remove the nonfree software from my phone.

To read the issues of freedom in Android in more detail, please check Richard Stallman’s article on this topic.

My experience of rooting

I have Samsung Galaxy S9+ phone and I searched on the internet for ROMs that can be installed in this device. I found the information on internet very confusing and unorganized. Since there was a risk of bricking my phone which would make it not usable, I thought of consulting someone who has installed custom ROM many times and can give some of their time to guide me. I consulted Abhas for this who is a hacker and has a lot of experience in installing custom ROMs. We were on a video call and Abhas gave me instructions.

The steps we did are listed here in this text file. Note that I used Debian, which is a GNU/Linux based operating system in my laptop. Depending on your phone model and laptop operating system, the steps mentioned at the URL will differ.

First, we tried installing Carbon ROM but it failed somehow. Then Abhas suggested me to try iode OS and I installed it. The XDA page for iode OS for my phone is here.

Check iode OS Screenshots taken by me after the fresh install (I may have installed a few apps before taking these Screenshots)

Important Note: After the install, the Jio sim does not work because iode OS (and many other custom ROMs) do not have Volte support.

For gaining root access, I flashed Magisk using TWRP.

After this, I learnt flashing a ROM myself and I successfully installed an older version of Lineage OS myself on Moto G4 Plus.

How much free software does iode OS have

iode OS is a free software operating system which does not have any Google apps pre-installed. Infact, the only nonfree software app pre-installed is Magic Earth. I immediately uninstalled that app. I suggest iode OS team to remove Magic Maps. If they would like to add any maps app, my suggestion to them is to add Organic Maps which takes data from OpenStreetMap.

The phone still contains nonfree firmware which is necessary for some hardware, like WiFi, to function properly, and the nonfree firmware itself is a serious issue in terms of user freedom. Even though the phone does not contain 100% free software, it is still a big step towards my freedom. There are no Google or Samsung apps in my phone which were pre-installed in the phone.

Nonfree software in iode OS: Magic Earth + nonfree firmware.

List of a few pre-installed software in the iode OS:

• F-Droid, modified to give priority to iode repository;

• QKSMS

• iode Browser

• microG

• pep

• Geometric Weather app

How to install a custom ROM

You can also install a custom ROM if you have an Android phone. If you have an iPhone, I don’t know anything about flashing a custom ROM there as of now. Perhaps ditching the iPhone is the only step towards freedom.

To find a custom ROM built for your phone, first check if Lineage OS has an official support for your phone. If Lineage OS is available for your device, then go for it. Also, check Lineage OS website for instructions to install custom ROM on your phone.

Steps for choosing a custom ROM:

1. Visit the XDA Forums.

2. Scroll down to All Categories section.

3. Choose your manufacturer from the All Categories list, for example, Samsung, Redmi etc.

4. Choose exact model of your phone, like Samsung Galaxy S9+. For my phone, this leads to this page.

5. Choose ROMs, Kernels from the list. This leads us to this page for my phone model.

6. Check for posts with OSS and OFFICIAL tags. Also check if gapps (Google Apps) are included or not. I suggest you to not include gapps.

If you click on a post, it will also contain instructions on how to flash a custom ROM.

Now the steps involved in installing custom ROM on your phone are:

1. Unlock the bootloader of the phone.

2. Run the phone in download mode and flash a recovery project like TWRP into the phone.

3. Use TWRP to wipe all the data– Dalvik Cache, system, cache, data. Flash the downloaded ROM image.

4. Reboot to recovery.

Depending on your device and the operating system in the laptop, the way to perform these steps would differ.

I know that these steps can be overwhelming for nontechnical users or users who haven’t done this before. Therefore, I am planning to make a video tutorial on rooting a phone in the future.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/firmware/ https://ravidwivedi.in/posts/firmware/ Mon, 06 Sep 2021 00:00:00 GMT Index

• What is firmware

• Problems with proprietary firmware

• Importance of Free Firmware

What is firmware

Firmware is a software which provides a way for hardware to interact with the operating system. It runs on a secondary processor and not on the CPU. For example the WiFi chipset will run this code directly instead of the main CPU. Firmware files aren’t used by the kernel, they’re loaded by the kernel onto other pieces of hardware.

Quoting Debian wiki on what is Firmware:

Firmware refers to embedded software which controls electronic devices. Well-defined boundaries between firmware and software do not exist, as both terms cover some of the same code. Typically, the term firmware deals with low-level operations in a device, without which the device would be completely non-functional.

For example, remote control of a television uses firmware to convert your button presses into infrared signals so that the television can understand. For more examples of firmware, please check this wiki page.

Let’s suppose you are able to boot and install a free software operating system, like any GNU/Linux based Operating System and you did not install any nonfree software. You might be tempted to say that you are running only free software but you might still be running proprietary firmware. For example, Ubuntu comes pre-installed with nonfree firmware. Depending upon your laptop, you might need to install nonfree firmware in addition to the operating system for some hardware to work properly.

Problems with proprietary firmware

The problem with propietary firmware is the same as any propietary software– users do not control it and we cannot trust it. It does not give users the freedom and therefore users do not have a defence against any of the bugs or intentional malfunctionalities being introduced in the system.

Quoting Mark Shuttleworth, the founder of Canonical which develops and maintains Ubuntu,

If you read the catalogue of spy tools and digital weaponry provided to us by Edward Snowden, you’ll see that firmware on your device is the NSA’s best friend.

Following are some known examples of insecurity issues due to bugs or adding intentional malfunctionalities at firmware level:

• A researcher broke into Apple’s firmware and caused the battery to overcharge

• NSA adds spyware into firmware of many devices.

• Wikileaks revealed that several CIA projects infect Apple Mac firmware (meaning the infection persists even if the operating system is re-installed). Basically, EFI/UEFI is essentially a backdoor for taking control of a computer without user’s knowledge. Further, the attacks described in the leaks are very widespread.

• A vulnerability in the EFI/UEFI firmware is used by CIA to write to NVRAM or persistent storage of the computer without the users knowing about it. It cannot be detected easily by the users and it can survive a complete OS reinstall.

• CIA documents leaked by the Wikileaks note that the UEFI level exploits can compromise the whole system. The ExitBootServices Hooking page notes, “At this point, you can do whatever you want.”

Keep in mind that if a backdoor is installed at the firmware level, it can gain full access to your computer. Such a control of the system means that the applications running on your machine are also fully compromised.

Importance of free firmware

In the previous section, we explored that the issue of firmware being propietary and hence, not controlled by the user, is not a light issue at all. It is a very serious issue and the well-known exploits in these firmware can make you a prey to NSA or any third-party spying without you even knowing.

The firmware should be free software as any software so that the users have freedom to detect and fix bugs, remove any backdoors put into the firmware, and other problems that can arise with proprietary firmware. For this, the manufacturers need to publish key technical specifications sufficient to write free firmware for their hardware. Even if you are running proprietary firmware and ditch Windows or MacOS or any proprietary operating system to switch to a free software operating system, like GNU/Linux, I welcome your move as one more step towards freedom.

A very important example of a firmware is BIOS, which starts your computer when you power it on. Libreboot and Coreboot are free software BIOS.

Liberated Computer (the laptop I use) runs on Coreboot and the only nonfree software it has is the Intel ME blob which is disabled.That means it runs purely free software.

The following devices have only free firmware installed at the time of shipping:

• Devices which can run Replicant OS (Note: You might have to use external hardware/dongles for WiFi, Bluetooth to work)

• Devices that have Libreboot support

• Devices which are RYF certified by FSF

Unfortunately, proprietary firmware is not even considered a problem by many people who care about software freedom. Perhaps there is a lack of awareness of the issue. If that is the case, then hopefully I have now made you aware of the issues with proprietary firmware.

(Credits: Thanks to Pirate Praveen for proofreading the article and correcting some factual errors.)

For further reading:

• Debian wiki.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/liberated-computer/ https://ravidwivedi.in/posts/liberated-computer/ Sun, 05 Sep 2021 00:00:00 GMT Update: Added some pictures on 29-April-2022.

What is Liberated Computer

Liberated Computer is a computer sold by LibreTech Shop based in Bangalore, India, which can run exclusively on free software (which means it respects user’s freedom and not that it is free-of-cost) and does not ship with any proprietary software installed. Liberated Computers are basically old and refurbished Lenovo’s Thinkpad laptops, modified in several ways so that they can run purely free software (check the section “How is an LC230 assembled?” in LC 230 docs). In addition, it respects user’s hardware freedom as well– you can do whatever you wish to do with the hardware, with no locks or constraints imposed by the manufacturer.

The computer is liberated by replacing the BIOS which won’t let a chip run which is not in its allow list. Further, the Intel ME backdoor has been disabled.

It has Coreboot whose code is 100% free software. The x230 BIOS contains the Intel management engine - which is then neutered using me_cleaner. That is a blob (A blob means a device driver whose source code is not published, only the binary is published; blobs are nonfree software) but that blob is inactive during the course of normal working of the operating system. There is no other blob in Liberated Computer. I will explain why free firmware matters in the next post. This makes Liberated Computer run on 100% free software.

Here is a screenshot of a tweet by Leah Rowe, the founder of Libreboot project, on Lenovo x230(Remember that Liberated Computer is just refurbished Lenovo’s x230) and Coreboot.

Why I bought Liberated Computer

I bought a Liberated Computer and the reason is simple: the attitude that freedom comes first and features are secondary. Free Software provides users the freedom to run, study, modify, improve, share the software. If the software lacks any of these freedoms, it is called nonfree/proprietary software.

I think free software is necessary for:

• privacy and security of users, though it might not be sufficient;

• learning and doing things ourselves;

• having a defence against the mistreatment by proprietary software.

There are many other reasons to use free software.

I also think that the freedoms provided by free software are every user’s rights. I am not of the opinion that we use any software that works, but the software should respect user’s freedom. Naturally, I would like to run only free software and eliminate all the nonfree software from my life.

My Purchase

I used to have Apple’s Macbook which is a very locked system and you don’t own it even after purchasing it. After I got convinced that software freedom is important, I tried to use only free software and avoid proprietary software as much as possible. However, a lot of free software for my use was not available for Macbook. And I doubt if Macbook can run without any proprietary software. Therefore, I was looking for a computer which can run exclusively on free software, without requiring any proprietary software. I knew that Librem laptops can run purely free software and they satisfied my requirement. I didn’t know about any Indian vendors selling such a laptop. One day, I asked in the FSCI matrix room whether there are Indian shops selling such a laptop. I came to know about Libretech Shop based in Bangalore which sells free software powered laptops and calls them ‘Liberated Computer’. A few months later, I decided to buy a LC 230.

I ordered LC 230 using the Mostly Harmless website, with 16 GB RAM, 480 GB SSD, and a new 6-cell battery.

Item Description	Amount (INR)
Base Price	₹ 27,000.00
RAM (16 GB)	₹ 3,500.00
SSD (480 GB)	₹ 2,500.00
New Battery (6-cell Battery)	₹ 3,500.00
Subtotal	₹ 36,500.00
Shipping (Flat Rate)	₹ 1,200.00
GST	₹ 6,786.00
Total	₹ 44,486.00

I placed my order on a Saturday. Abhas shipped it to me, three days later, on Tuesday via DTDC priority shipping and I received the laptop on Thursday.

My experience with LC 230

The laptop had KDE Neon pre-installed but Abhas (the owner of the shop where I purchased Liberated Computer) suggested me to re-install any OS of my choice on my own. I re-installed KDE Neon using a bootable USB. I explored KDE Neon and liked it. I found it user-friendly and got comfortable in a few hours.

Then I booted PureOS, Ubuntu, Kubuntu, Manjaro KDE, Solus Budgie etc. Finally, I settled on Debian 11 KDE and I am using it as my main OS right now.

The new battery lasts around 5 hours from full charged state to zero.

The low speaker volume is somewhat offputting. Instead of a backlit keyboard, it has a Thinklight which flashes light on the keyboard to work in the dark.

It has hardware kill switches for microphone, audio and wireless connections. It has a webcam too.

Switched to GNU/Linux

Finally, I switched to GNU/Linux. The GNU/Linux operating systems are free software and are fully under user’s control and this is exactly my reason for the switch. Nonfree software macOS and Windows are full of trackers. Debian, on the other hand, does not have any trackers or spywares. It respects my freedom and privacy.

Happy Hacking!

Pictures

Following are some pictures of Liberated Computer taken by me. All of them are released under CC-BY-SA license.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/meeting-rms/ https://ravidwivedi.in/posts/meeting-rms/ Sat, 31 Jul 2021 00:00:00 GMT I am writing about a meeting which happened more than 5 years before writing this article. This is the best recreation from memory I came up with.

At the time this happened, I was a student at Acharya Narendra Dev College (also known as “ANDC” in short), New Delhi. On the 1st of March 2016, the principal of my college, Dr. Savithri Singh, was addressing all the students in a common hall (for some reason I don’t remember). During the address, she told us that the founder of Linux was visiting our college that day. She invited interested students to meet him to her office at 4 PM. I was sitting with my friend, Shivam Rai who was excited for the meeting. I am from mathematics background and hardly knew anything about Linux. I had heard the word “Linux” and knew that it is a kernel of some operating system. So, I wasn’t very entusiastic.

Around 4 PM, I was a bit tired and wanted to go back to my room to take a nap. I was not completely sure about the point of the meeting. Furthermore, I thought they discuss programming or coding like mathematicians discuss theorems and proofs. I almost forgot about it. However, my friend reminded me of the meeting, so I joined him.

We went to the principal office and after waiting for some time, the visitor arrived along with a person from Kerala. My friend and I shook hands with the visitor, and he said, “Hi, my name is Richard Stallman”. We also introduced ourselves. The person from Kerala also introduced himself but unfortunately, I forgot his name. Immediately, Richard Stallman asked us a question in a joking manner, “If I were to start a company in India, what should I name it?”. We were silent. Then he said, “Mahadeva” followed by a big laughter. It turned out that, even though the principal invited the whole college to meet Richard Stallman, only two of us actually went to meet him. Next, since the principal said that he is a founder of Linux, we started the discussion something along the lines of “You are the founder of Linux?” Listening to this, he got angry and said, “You should ask that question from Linus Torvalds.” I was confused at this point, and so was my friend.

Then we asked, “What do you work for?”. He responded, “I work for GNU.” I asked, “What’s that?” He said, “GNU’s not Unix.” I got very confused at this point as I didn’t know what’s GNU, what’s Unix, and we already pissed him off by associating him with Linux.

Then, he said, “I work for free software.” This added to my confusion as I thought why would someone work for free-of-cost software. Then he explained what is free software, “Free Software is software that gives users four freedoms.

Freedom 0 is the freedom to run the software as you wish.

Freedom 1 is the freedom to study and modify the program. You need to have the source code of the program to exercise this freedom.

Freedom 2 is the freedom to share the program.

Freedom 3 is the freedom to share your modified versions.

The ‘free’ does not mean free of cost. It means freedom. Think of it like free speech and not free beer.”

At that point, I only knew that a lot of programs do not have freedom 1. I didn’t know that a lot of software do not give users the freedom to share or run. So naturally, I didn’t understand why he has to emphasize these freedoms and why they are important. Also, I used to think every software have freedoms 0,2,3. I remember having a (wrong) notion that the freedom to study and modify the source code is for the programmers only.

Then, he said, “Any software which does not give users these four freedoms is called a nonfree/proprietary software. A proprietary software is under developer’s control, and you are at the mercy of the developer. A free software is controlled by its users. I use only free software and reject all proprietary software and, you should do it too.”

His voice was quite passionate when he was telling us those things. Further, he told us that Microsoft Windows is an example of a proprietary software and mentioned that a backdoor was found in Windows in the past. I didn’t know what backdoor means and I don’t remember if he told us that.

He had hearing problems, so we had to be loud while interacting with him. My friend then asked about Apple’s software. Something like “Is Apple’s software secure?” Stallman replied something related to Apple’s locking of user and Apple’s software being nonfree. Bu I forgot a lot of at he said.

I remember that he also said that Google services are surveillance systems, and he does not use them. He insisted that we should also refuse to use them. I remember him mentioning that NSA spies on Americans as well as other countries’ citizens using Google services like Gmail, Google Maps. Poor me, I didn’t know what is NSA.

He also mentioned that Google Maps does not load if he disables the Javascript code sent by the site. Earlier, it used to load without Javascript. He kind of implied that sites should load without Javascript. And he probably gave a few more examples when disabling Javascript does not load the website. At that time, I didn’t understand why would someone disable Javascript when visiting a website.

Then, our principal reminded us to take photos with Richard Stallman. Stallman agreed with the condition that we never upload his photo on Facebook, Instagram or WhatsApp, because that company is a surveillance engine, and we should not feed it. We agreed with the condition and clicked photos with him.

Richard Stallman was then treated with Dal Vada from our college canteen, and he really liked that.

This meeting planted a few questions in my mind about the software I was using. At that time, I didn’t have adequate political understanding to get his point. I also found it a bit sad that the person is known for “Linux” and “Open Source”, even though he despises those labels.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/how-i-removed-gapps/ https://ravidwivedi.in/posts/how-i-removed-gapps/ Tue, 27 Jul 2021 00:00:00 GMT In this post, I will write how I deleted (yes, deleted, uninstalled, gone! not disabled) Google apps, Samsung apps and Facebook app from my phone. These apps are pre-installed and they only give an option to “Diasble” these apps which does not do anything, it only stops showing them in the app drawer. These apps are dangerous to user’s privacy and record their whole lives through their phones. You do not need to know anything about the command line to follow this guide. showTableOfContents: true

Sahil’s blog post on Mobile Made More Private was the one which guided me in this process. Be sure to check out that for more details and the adb commands cheat sheet that Sahil mentions in that post.

A fair warning and comment from Chris Titus Tech:

It is dangerous to remove system apps that you don’t see in the app drawer. It can lead to you doing a full system recovery.

On Non-Rooted phones some applications are installed as root and error with “[DELETE_FAILED_INTERNAL_ERROR]”. Type this to bypass:

adb shell pm uninstall –user 0

So, it is a “Do at your own risk” article.

How I did it

I have a Samsung Galaxy S9+ phone with Google and Samsung’s nonfree software installed in it. I did not yet root my phone to install a custom ROM. I have a Macbook which does not allow me to connect my Android using a USB cable, a step which is essential to run command line in your android to uninstall these apps.

So, I connected my phone to a laptop in which Windows 10 was installed. The steps are similar and commands are same (except for the grep command which works in GNU/Linux but not in Windows, which I will mention) if you using GNU/Linux or macOS.

These are the steps I followed in Windows 10:

Step 1 : Install chocolatey, a freedom-respecting command line program to download software in Widnows. The steps to install Chocolatey are here. You don’t need to visit that link, I will write all the steps for you.

Below are the steps for installing Chocolatey on Windows.

Step 1a : Press windows button on your keyboard and search for ‘PowerShell’. Right click on PowerShell and click on an option which says run it in the administrator mode.

Step 1b : In the PowerShell, run the command (which means, copy the following command and paste it in PowerShell and press enter on Keyboard):

Get-ExecutionPolicy

If it returns Restricted, run the command:

Set-ExecutionPolicy AllSigned

Step 1c : Run the following command:

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

If you do not get any error message, congratulations! Chocolatey is installed in your system.

Type choco -? in the command line and press enter. If there is no error, it means that Chocolatey has been installed in your system.

Step 2 : We will install ADB Platform Tools. To install, run the command:

choco install adb

Step 3 : Setting up your Android phone

3a : Open Settings on your phone, and select “About”.

3b : Tap on “Build number” seven times.

3c : Go back, and select “Developer options”.

3d : Scroll down, and check the “Android debugging” or “USB debugging” entry under “Debugging”.

3e : Plug your device into your computer.

3f : On the computer, open up a terminal/command prompt and type adb devices.

3g : A dialog should show on your device, asking you to allow usb debugging. Check “always allow”, and choose “OK”.

Step 4 : Run the command:

adb devices

This command should list the Android phone you connected to your computer. If it lists your phone, you are ready to proceed. Otherwise, your phone isn’t connected and not ready to use adb commands.

Step 5 : You can list all the packages of Google in your phone by using the command:

In Windows: adb shell pm list packages | findstr google

In GNU/Linux or macOS: adb shell pm list packages | grep google

The following image has commands that can be used to uninstall Google apps.

Uninstalling the apps

Let’s try and uninstall the YouTube app. Check the image above for the package name. The package name for YouTube is com.google.android.youtube. The following command should be the one to uninstall YouTube (same command for Windows, macOS, GNU/Linux):

adb shell pm uninstall -k --user 0 com.google.android.youtube

If the command returns the message saying: Success, then you have successfully uninstalled YouTube.

Similarly, you can uninstall any other app using this command by replacing com.google.android.youtube by the package name of the app.

For removing Google Play Store, I used the following command:

adb shell pm uninstall -k --user 0 com.android.vending

For me, it was no problem uninstalling Google Play Store but I have heard it can be risky.

I have listed all the commands that I tried in my phone in adb-commands-ravi.txt. Feel free to use some or all of these commands.

A trick to know the app ID for most of the apps is to visit their Google Play Store page. For example, visit the Facebook app on Google Play Store and look at the URL. Copy the part after id= which is com.facebook.katana.

Therefore, the command to uninstall Facebook will be:

adb shell pm uninstall -k --user 0 com.facebook.katana

That’s it for now.

Happy Hacking with (hopefully) less amount of tracking:)

]]> Ravi Dwivedi https://ravidwivedi.in/posts/xmpp-guide/ https://ravidwivedi.in/posts/xmpp-guide/ Sat, 24 Jul 2021 00:00:00 GMT This is a guide to set up an account on XMPP, which is a federated chatting system.

You can watch this video tutorial to create an XMPP account without giving your phone number to the service:

I will use the freedom-respecting software Conversations app for demonstration. Conversations app is available for Android only. Conversations is a paid app on Google Play Store. If you would like to support the development of Conversations app, then you can download it from Google Play Store. Otherwise, you can follow the steps below to download Conversations app free of cost. If you would like to download a free-of-cost XMPP app from Play Store, then check out Blabber app.

Currently, Blabber app is very good for XMPP. It has some more functionality than Conversations app and these features are usually requested by users in the Conversations app.

Some examples of XMPP apps for other platforms are- Dino for GNU/Linux, Gajim for Windows and GNU/Linux, Monal for macOS and iOS. You can choose any XMPP app, and this guide probably works for all of them. A list of XMPP apps is here and all of them support OMEMO encryption by default.

You don’t need to give any personal details, like phone number or email, to create an account. If you would like to register an XMPP account using your phone number (similar to how you set up in WhatsApp), then you can try Quicksy app, which respects your freedom as you can still connect to people using any XMPP app and XMPP service provider (remember that XMPP is federated!).

Steps to set up an account on Conversations app:

Step 1: Download F-Droid from here.

F-Droid is a repository of free software apps. It is a privacy respecting alternative of Google Play Store. One difference is that F-Droid only contains apps which are freedom-respecting (they respect all four freedoms mentioned here) and verified by the F-Droid community for malicious features.

F-Droid will also ensure that your app receives all the future updates. F-Droid community also builds apps from the source code which ensures that the source code indeed matches the app you download in your system.

Step 2: Search for “Conversations” app in F-Droid and install it.

Step 3: Register an XMPP account on any xmpp server. You can choose an XMPP service provider from this URL. It will give you a list of 5 servers that you can register on, and then will give you 5 different random servers upon refreshing the page.

Step 4: Open the Conversations app and then click on ‘I already have an account’ option.

Step 5: In the XMPP address option, put your xmpp address, which will be of the format username@domain – depending on which site you used to sign up. Click ‘Next’.

Step 6: You can add a profile picture if you wish and then click ‘Publish’. If you do not want to add a profile picture, select ‘Skip’.

Step 7: Conversations app will ask you permission to access your contacts. You can click on ‘Deny’. And it will then ask to stop battery optimization for the app. You can enable Conversations to run in the background.

Step 8: Conversations app will then ask you to stop battery optimization for the app. I suggest you to allow Conversations to run without optimizing the battery. The developer of Conversations app has explained how this permission has virtually zero impact on battery life.

Step 9: To add a contact, press the ‘+’ button at the bottom of the Conversations app and click on ‘Add Contact’.

Step 10: Type the XMPP address of the XMPP user you want to contact. In the image, I have entered my own XMPP address - ravi@poddery.com - as an example.

Step 11: Just send an introduction message to your friend to exchange your OMEMO keys.

Congratulations on setting up your XMPP account. All the messages will be OMEMO encrypted by default – means that only you and the person you are exchanging messages with can read them. Conversations app also support voice call and video call to any XMPP app that supports the same.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/fsci-jmc/ https://ravidwivedi.in/posts/fsci-jmc/ Wed, 21 Jul 2021 00:00:00 GMT showTableOfContents: true Free Software Community of India(FSCI) is committed to promote software freedom and make privacy accessible to common people. Campaigning and raising awareness for privacy and urging people to not use Google/Facebook/Zoom and other proprietary services is not enough, users must also have alternatives to these services which were pretty much turned into necessity (for privileged people) especially in the pandemic time. Jitsi is a freedom-respecting video-conferencing software which gives users the control over the software. Jitsi allows anyone to run their own server, allowing them to fully control their means of communication. Since it is not feasible for everyone to run their own Jitsi server, FSCI decided to work on running their own server which would be open to all, can be used by anyone for their meetings, funded by voluntary donations and must respect user’s privacy.

To run a server funded by donations, we needed volunteers for: running and maintaining a Jitsi server, writing script for the fundraiser video, asking for donations in the fundraiser video, editing the fundraiser video, maintaining website, giving a bank account to accept donations, replying to emails regarding queries, taking initiative and coordinating all this.

I think it was around November 2020 when Sahil told me about FSCI’s initiative to run a Jitsi server and I agreed to volunteer for asking for donations in the fundraiser video. Arun Mathai guided me in the process. We had meetings and we rehearsed the script a few times and assigned the lines. I had some problems shooting the video of myself speaking in the camera. Talking to myself alone on camera is not exactly my plus point. After several attempts, I could make a clip which was fine to be included in the fundraiser video (thanks to mom for shooting the clip).

We thought it would be good if there is diversity among the people asking for donations in the fundraiser video. So, I asked my friend Ashutosh, who is also inclined towards free software, if he can help. Then he discussed the fundraiser with his friends and two of his friends volunteered to appear in the video to ask for doantions. This opportunity raised awareness about free software and existence of community-powered services among his friends – which furthers our goal. I also asked my friend Sayan for recording the Bengali snippet of the video and he agreed to volunteer for the same.

It took some months after that (we did all of the above by the end of December 2020) to finally release the website and the Jitsi server acutally operational. After the release, we, at FSCI, use meet.fsci.in for all our meetings. I use this server for all my personal meetings.

I am glad that the fundraiser is going strong – as of writing, we have raised 57% of the funds needed to run the service for 2 years. As of now, I think that drafting the privacy policy is a pending major task which I will try to do as soon as possbile.

You are invited to use the Jitsi server at meet.fsci.in for your personal meeting needs without worrying about someone prying on you. You can also host your classes. The server can handle around 30 participants in a meeting. If you would like to support the service, you can donate by visitng here or volunteer to help us in running the service by contacting FSCI. The transparency is ensured by listing all the contributors and their donation amounts publicly (you can be an anonymous contributor).

Please send your comments/suggestions on the fundraiser/meet server to FSCI (the contact option is at the bottom of the link.

Thanks to all the volunteers and contributors for making meet.fsci.in possible.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/apple-trap/ https://ravidwivedi.in/posts/apple-trap/ Tue, 13 Jul 2021 00:00:00 GMT Apple is a company which claims that protecting users’ privacy is one of their core values. Recently, they announced a new software update to their iOS with the idea that the apps made by independent developers have to request explicit permission from users before tracking them.

That means Apple cares about users’ privacy.

Or do they?

Sure, they are not involved in targeted advertising, and they don’t earn money from your data and therefore, do not track you, right?

And they encrypt all contents of iOS devices, making it impossible for third-parties to access your data, right?

They are working day and night to make your life better, convenient and at the same time, defend your privacy. What more can we ask for?

Well ….. we don’t know if any of these claims are true

because Apple’s iOS is a nonfree/proprietary program, and therefore these claims cannot be independently verified. Apple does not give its users the freedom to study how iOS works– it does not provide its source code.

Even if Apple’s claims are true, this only means you get privacy from others and not from Apple. They might be encrypting all your contents stored in iOS, we don’t know, but they have also installed backdoors. This means Apple can make changes in your devices remotely without your permission. They can add any malicious features in the software at any time without you knowing. The backdoors were found in macOS as well as in the iOS. That means your computer isn’t yours.

While Apple says that it respects its user’s privacy, their track-record says exactly the opposite.

Snowden’s documents show that Apple is a part of NSA’s global surveillance program.

In China, it stores all user data on servers controlled by the Chinese government.

All the search queries made in macOS’ spotlight are send to Apple.

Apple uploaded private files of its user to iCloud without consent.

You remember who is Siri? Siri is Apple’s voice assistant. It is Apple’s voice assistant, not your voice assistant. You might think that Siri listens to you only when prompted, but it turns out that it listens and records all the time. Apple also hires people specifically for listening to users’ Siri recordings.

You can check out more cases of surveillance by Apple by visiting this link.

Cory Doctorow argues that surrendering your autonomy by moving to Apple’s fortress has the same problem as all benevolent dictatorships: it works well, but fails badly. Let me quote Cory Doctorow from this article,

Apple rightly points out that the world is full of bandits who will steal your data and money and ruin your life, and it holds itself out as your protector …. But when Apple sides with the bandits, the walls that once protected you now make you easy prey.

Overall, Apple claims itself to be the guardian protecting its users’ privacy, while their track record is exactly the opposite, they increased surveillance on users with the latest macOS update, they won’t allow third-party repair shops but their “authorized” technicians themselves leak users’ data. Apple’s privacy claims, at their best, only give users privacy from third-parties. That is what their privacy trap is– Redefine privacy to mean privacy from others and earn goodwill by advertising that they care about users’ privacy. Surveillance is just a symptom of proprietary software. The bigger problem is that Apple keeps tight control over iDevices and users don’t control them.

The only software we can trust is free software, which respects user’s freedom. Users have control over the software if it is free software. If the software is free, users can check for malicious code and remove the malicious functionalities.

A nonfree software, on the other hand, is malware. Apple’s iOS is malware because it is nonfree software. It is a black box. We never know what it is up to, so any trust on Apple respecting user’s privacy is a blind trust.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/accessing-youtube/ https://ravidwivedi.in/posts/accessing-youtube/ Mon, 12 Jul 2021 00:00:00 GMT YouTube is a platform where people upload and watch videos. I don’t think YouTube needs any introduction among the internet users. The thing that is not usually mentioned about YouTube is that it tracks its users. Further, YouTube’s algorithms distort the truth, it censors works that are fair use under the copyright law and doesn’t let you download videos in your device, even the ones you purchased.

An analysis of YouTube’s privacy policy is here.

In short, it is not YouTube, it is TheirTube. Knowing someone is always watching you has inhibiting effects. I don’t want to get tracked or get manipulated by YouTube’s algorithms. Fortunately, YouTube is accessible using free software in ways which respect your privacy. I will list some ways to watch YouTube videos using workarounds. All the software mentioned here are free software(which means they respect your freedom). Please read this article to understand why we must insist on free software.

In Desktop, YouTube videos can be watched using Invidious. You can visit any of the Invidious instances here from any browser and search for the video you would like to watch. Invidious also allows you to visit via Tor which hides your IP Address. You can use Privacy Redirect plugin in Firefox and Tor to automatically redirect YouTube links to Invidious. This plugin also redirects Twitter, Instagram, Reddit, Google Maps, Google Translate links to their privacy-respecting front-ends. You can also check out a bunch of other freedom-respecting software that you can use to watch YouTube videos.

Piped is a free software which acts as a frontend to YouTube. You can watch YouTube videos using Piped in your browser by visiting this URL. Piped does not expose your IP address to YouTube.

FreeTube app can also be used to watch YouTube videos. The app is available for all the GNU/Linux distros, macOS and Windows. FreeTube respects your privacy as well. Please read privacy section of their FAQ for details.

Youtube-dl is another software you can use on Desktop to watch videos. It is a command-line program that downloads videos from YouTube in your system. It is my favorite, and I fell in love with it when I realized how capable it is. It is very powerful, and it supports downloading videos from many other sites as well. You can also download audio files by extracting audio from YouTube videos using YouTube-dl. It supports downloading playlists as well. I plan to write a blog in future on using command-line to download using youtube-dl for beginners.

For Android, Newpipe app can be used to watch YouTube videos. Newpipe app shows no ads, does not track you, can run in background, can download audio/video files in your phone. Newpipe can also stream videos which are being live-streamed on YouTube. In addition to YouTube, you can access SoundCloud, media.ccc.de, PeerTube instances ,Bandcamp using NewPipe.

For iOS, I don’t know any workarounds. If you know of any, please let me know. iOS is, in general, very hostile to user freedom, so the only way to freedom is to avoid iOS.

While these workarounds work well, we cannot rely on Google to let them work. The workarounds do not really solve the problem. The real solution is that the videos are posted on a platform which respects user’s freedom(which is a bit more complicated issue than the freedom of software installed in your system). You can upload videos on Peertube or your own website to respect user’s freedom and allow them to download the videos as well. I have a channel on Peertube on the instance set up by FSCI. You can set up a donation based model to earn money from your videos. Users can send you money directly in your account for your work. You can also earn money by selling merchandise. For this model to be successful, we need more people to join in rejecting YouTube and similar tracking services.

]]> Ravi Dwivedi https://ravidwivedi.in/posts/chatting-apps/ https://ravidwivedi.in/posts/chatting-apps/ Mon, 12 Jul 2021 00:00:00 GMT Which chatting app do you use to contact your loved ones?

Do you use WhatsApp?

Do you use Telegram or Signal?

What if you do not agree to the terms and conditions or privacy policy of these apps?

Well, either you need to accept their terms or switch to any other app and do the hard work of convincing every contact to move to your new chat app whose terms you can agree. What if you are a student and all the important notifications are sent to the WhatsApp group? Would you convince your school to avoid WhatsApp? What if they don’t care? Now you are forced to be on WhatsApp to make sure you don’t miss any important updates. So, when you click on ‘I Agree’ to the document which says, “We will put you in surveillance. It has a lot of benefits. Your life will be very convenient”, do you really agree, or you just gave in to social pressure? Or you did not care? Do you really have a choice?

So, how do we control our means of communications? What does controlling our communications mean? If we control our means of communications, can we ensure privacy as well? I will go into details of what I mean when I say that the users control the software. If the users have the freedom to run, copy, distribute, study, change and improve the software, then the users control the software. Such a software is called free software, where ‘free’ refers to freedom and not to price. In this article, ‘free’ refers to freedom and never to price. I suggest you to read this article to understand why these freedoms are important and how this gives users control over the software. Examples of free software(freedom-respecting) chatting apps are Telegram, Signal, Quicksy, Element etc. If users lack any of these freedoms, then the software is called nonfree/proprietary and such a software cannot be trusted by the user. WhatsApp is an example of nonfree/proprietary software.

Chatting apps usually have two components: 1. The app that you install on your device; 2. A server (we will call it a service provider)which transfers the messages from the sender to the recipient. If you control only the software, the service provider still has the power to impose unjust conditions on you. In the above-mentioned example, Signal is a free software which includes the freedom to modify the code but when a project modified the Signal app code, Signal refused to allow them to connect to their server or federate with any other server. This is not true freedom. So, to control our chatting system, software must be free but that is not enough.

Therefore, for full control, we need to have federated chat systems – to allow users registered on different service providers to communicate with each other - for instance a mail server run by Google federates with a mail server run by Microsoft when you send email from @gmail.com to @hotmail.com. So you can choose a free software and a trusted, community-run service-provider, and this is what I mean by having control over our communications. This control is collective control by community. Examples of such systems are Matrix and XMPP. Federation answers the question raised earlier: What to do if the service provider imposes terms and conditions you do not agree with? You can switch to another service provider or you can be a service provider and still communicate with your contacts. You don’t need to convince them to switch to a new provider. Examples of matrix apps are- Element, Nheko, Fluffychat etc. Examples of XMPP apps are- Conversations, Dino, Gajim, Siskin IM etc. Make sure that the app supports end-to-end encryption–which means only the sender and recipient can read the messages.

Quicksy app is at the intersection of freedom and convenience. It registers the account with a phone number, which makes the app convenient and easy to use. It federates with XMPP so no one is forced to use Quicksy and so other people can use some other XMPP app(like Conversations) which do not require any personal details to create an account.

To get started with XMPP, you can use this guide or use Quicksy app if you don’t mind registering with your phone number.

Telegram and Signal apps are freedom-respecting but since they are not federated, the overall chatting system is not freedom-respecting.

Another way to freedom is to use freedom-respecting encrypted messaging apps which do not involve any servers. These are called peer-to-peer apps. The downside here is that both the users need to be online at the same time to exchange messages. You can choose to run the app in background to receive messages. Examples: Briar, GNU Jami, Tox.

TL;DR : Free Software app + Federated chat systems like Matrix or XMPP and free software peer-to-peer apps like Briar give users full control over their communications and therefore you can ensure privacy. Nonfree/proprietary software control the users and therefore cannot be trusted for privacy.

Further Reading:

• FSF India’s article on the same.

• It’s all about choices and control.

• Instant Messaging: It’s not about the app - XMPP provides sovereignty of your communication.

• How to ensure your Instant Messaging solution offers users privacy and security.

]]> Ravi Dwivedi https://aboobacker.in/2021/06/28/bootsnap-and-spring-understanding-rails-boottime-optimisations.html https://aboobacker.in/2021/06/28/bootsnap-and-spring-understanding-rails-boottime-optimisations.html Mon, 28 Jun 2021 08:33:00 GMT Ripper.tokenize code => ["def", " ", "add", "(", "x", ",", " ", "y", ")", "\n", "x", " ", "+", " ", "y", "\n", "end", "\n"] And see the parsed form using ripper sexp irb(main):042:0> pp Ripper.sexp(code) [:program, [[:def, [:@ident, "add", [1, 4]], [:paren, [:params, [[:@ident, "x", [1, 8]], [:@ident, "y", [1, 11]]], nil, nil, nil, nil, nil, nil]], [:bodystmt, [[:binary, [:var_ref, [:@ident, "x", [2, 0]]], :+, [:var_ref, [:@ident, "y", [2, 4]]]]], nil, nil, We can also take look at how the Instruction sequence(yarv code) looks like irb(main):052:0> puts RubyVM::InstructionSequence.compile(code).disasm == disasm: #@:1 (1,0)-(3,3)> (catch: FALSE) 0000 definemethod :add, add ( 1)[Li] 0003 putobject :add 0005 leave == disasm: #:1 (1,0)-(3,3)> (catch: FALSE) local table (size: 2, argc: 2 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 2] x@0 [ 1] y@1 0000 getlocal_WC_0 x@0 ( 2)[LiCa] 0002 getlocal_WC_0 y@1 0004 opt_plus 0006 leave ( 3)[Re] => nil Explaining the terms in the parsed and compiled code is beyond the scope of this blog post, and I recommend reading Ruby Under a Microscope. Newer versions of ruby also allows us to compile the code to binary and execute them later. it is experimental and platform dependent. ? cat example.rb number = 23 puts number + 23 ? ruby -e "File.write('example.bin',RubyVM::InstructionSequence.compile_file('example.rb') .to_binary)" ? cat example.bin YARB@ ?x86_64-darwin18%?#?%?gw numberE+Eexampleputs?????????% irb(main):018:0> RubyVM::InstructionSequence.load_from_binary(File.read('example.bin')).eval 46 That is it for the ruby compilation process for now. Now let’s how the ruby’s require method works. Contrary to what I thought initially, require is not a keyword in ruby, it is a method from ruby’s Kernal module. Let’s look at the overly simplified version of require method for our context. def require(file_name) eval File.read(filename) end Two main issues with this implementation are Requiring same file again will load the file again Only absolute paths are supported We can fix these issues in the following ways $LOADED_FEATURES = [] def require(filename) return false if $LOADED_FEATURES.include?(filename) eval File.read(filename) $LOADED_FEATURES << filename end $LOAD_PATH = [] #$LOAD_PATH = += gems_path + stdlib path + application code paths def require(filename) full_path = $LOAD_PATH.take do |path| File.exist?(File.join(path, filename)) end eval File.read(full_path) end While above code snippets are dummy implementations, Ruby actually uses the constants $LOADED_FEATURES and $LOAD_PATH for the same use case. Here is a stat from one of our app for reference irb(main):054:0> $LOADED_FEATURES.count => 6552 irb(main):058:0> $LOAD_PATH.count => 779 Another important method we need to recall is the fork system call in POSIX systems. fork allows the OS to create a new process as the child process with the same memory space. Modern hardware architectures like x86 allows the OS to optimise the fork with a mechanism called copy on write. In short it is very cheap to create a forked process of an app than loading that app from scratch. Now let’s look at the gems from the title. Spring is a rails only tool to speedup development and test environments. It creates your app process in the background for development and test environments and acts as a server. When you run a process like bundle exec rails server or bundle exec rspec it sends the data like command, ENV values, arguments etc to spring server, and spring server will fork the server process and run the task. Since the server process already loaded the app, the loading time of the app will be negligible and your task will run faster. But when the code changes, the server process needs to update and it may fail due to various reasons, like adding new directories. That is why you ended up having to run spring stop manually or restart the system to make the application behave as expected. The serve method in Spring is provided as reference. You can see zipping io values, fetching arguments, env etc from client, forking the new process etc in the below snippet. def serve(client) log "got client" manager.puts _stdout, stderr, _stdin = streams = 3.times.map { client.recv_io } [STDOUT, STDERR, STDIN].zip(streams).each { |a, b| a.reopen(b) } preload unless preloaded? args, env = JSON.load(client.read(client.gets.to_i)).values_at("args", "env") command = Spring.command(args.shift) connect_database setup command if Rails.application.reloaders.any?(&:updated?) Rails.application.reloader.reload! end pid = fork { Process.setsid IGNORE_SIGNALS.each { |sig| trap(sig, "DEFAULT") } trap("TERM", "DEFAULT") unless Spring.quiet STDERR.puts "Running via Spring preloader in process #{Process.pid}" if Rails.env.production? STDERR.puts "WARNING: Spring is running in production. To fix " \ "this make sure the spring gem is only present " \ "in `development` and `test` groups in your Gemfile " \ "and make sure you always use " \ "`bundle install --without development test` in production" end end ARGV.replace(args) $0 = command.exec_name # Delete all env vars which are unchanged from before Spring started original_env.each { |k, v| ENV.delete k if ENV[k] == v } # Load in the current env vars, except those which *were* changed when Spring started env.each { |k, v| ENV[k] ||= v } # requiring is faster, so if config.cache_classes was true in # the environment's config file, then we can respect that from # here on as we no longer need constant reloading. if @original_cache_classes ActiveSupport::Dependencies.mechanism = :require Rails.application.config.cache_classes = true end connect_database srand invoke_after_fork_callbacks shush_backtraces command.call } disconnect_database log "forked #{pid}" manager.puts pid wait pid, streams, client rescue Exception => e log "exception: #{e}" manager.puts unless pid if streams && !e.is_a?(SystemExit) print_exception(stderr, e) streams.each(&:close) end client.puts(1) if pid client.close ensure # Redirect STDOUT and STDERR to prevent from keeping the original FDs # (i.e. to prevent `spring rake -T | grep db` from hanging forever), # even when exception is raised before forking (i.e. preloading). reset_streams end Bootsnap is a gem released by shopify by extracting boot time improvements they made in their app. We can categorise them into two parts Path prescanning Kernel#require and Kernel#load are modified to eliminate $LOAD_PATH scans ActiveSupport::Dependencies.{autoloadable_module?,load_missing_constant,depend_on} are overridden to eliminate scans of ActiveSupport::Dependencies.autoload_paths. Compilation Caching RubyVM::InstructionSequence.load_iseq is implemented to cache the result of Ruby bytecode compilation YAML.load_file is modified to cache the result of loading a YAML object in MessagePack format (or Marshal, if the message uses types unsupported by MessagePack) If you look at the pseudo code above to demonstrate the LOAD_PATH behaviour, you will see that we need to check file existence every time we do a require, which is an io operation and not very cheap to perform. What if we can do something like this? def require(filename) if $CACHED_PATH[file_name] full_path = $CACHED_PATH[filename] else full_path = $LOAD_PATH.take do |path| File.exist?(File.join(path, filename)) end end eval File.read(full_path) end load path for a library is not something that changes very often, especially for gem paths and standard library paths, bootsnap caches them to save redundant file checks. Not that cache duration and expiration vary with files, so caching them in a constant won’t work. CACHED_PATH is just for reference and not used by the gem. Another important optimization by bootsnap is the compilation cache. We covered the ruby compilation process in the beginning and saw that every single file needed to go through the compilation process every single time they got called. Bootsnap addresses this by caching yarv code(instruction sequence) and compiling the code only if the code changes. There are also other optimised YAML loading by caching in an optimised format. overall Bootsnap gives impressive benefits as it works on production as well. We got about 30% reduction in boottime for our production ecommerce app. Note: This was originaly presented in internal tech talk in Sephora and made some tweaks to the content for wider audience.]]> Being the default the gem file entries in a rails project, you may have encountered spring and bootsnap. Here we are going through how they work and how they can improve boot time of your applications and why you get some weird errors in the app(Looking at you Spring).

Before diving into details, let’s go through some fundamentals to understand things better. Traditionally there were two kinds of languages, Compiled languages and interpreted languages. Compiled languages convert the code into binary format and that binary gets executed as the program. C, C++, Java etc are examples for this pattern. Another set of languages which were primarily designed for scripting read individual lines in the code and converted to machine instructions on runtime.

With ruby 1.9, a new virtual machine replaced the interpreter in ruby which was called Mat’s Ruby Interpreter(MRI). With a new virtual machine. Here instead of directly executing the code, ruby code will be converted to an intermediate representation called instruction sequence and the YARV virtual machine will execute them.

You can refer to the following diagram for reference.

Credit: Ruby under a microscope

There are also bulletin tools to inspect what is happening in the process, let’s check what happens for a 3 line code.

require 'ripper'
require 'pp'
code = <<CODE
def add(x, y)
  x + y
end
CODE

Let’s split the code into tokens using ripper utility

irb(main):043:0> Ripper.tokenize code
=> ["def", " ", "add", "(", "x", ",", " ", "y", ")", "\n", "x", " ", "+", " ", "y", "\n", "end", "\n"]

---

And see the parsed form using ripper sexp

irb(main):042:0> pp Ripper.sexp(code)
[:program,
 [[:def,
   [:@ident, "add", [1, 4]],
   [:paren,
    [:params,
     [[:@ident, "x", [1, 8]], [:@ident, "y", [1, 11]]],
     nil,
     nil,
     nil,
     nil,
     nil,
     nil]],
   [:bodystmt,
    [[:binary,
      [:var_ref, [:@ident, "x", [2, 0]]],
      :+,
      [:var_ref, [:@ident, "y", [2, 4]]]]],
    nil,
    nil,

We can also take look at how the Instruction sequence(yarv code) looks like

irb(main):052:0> puts RubyVM::InstructionSequence.compile(code).disasm
== disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(3,3)> (catch: FALSE)
0000 definemethod                           :add, add                 (   1)[Li]
0003 putobject                              :add
0005 leave

== disasm: #<ISeq:add@<compiled>:1 (1,0)-(3,3)> (catch: FALSE)
local table (size: 2, argc: 2 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 2] x@0<Arg>   [ 1] y@1<Arg>
0000 getlocal_WC_0                          x@0                       (   2)[LiCa]
0002 getlocal_WC_0                          y@1
0004 opt_plus                               <calldata!mid:+, argc:1, ARGS_SIMPLE>
0006 leave                                                            (   3)[Re]
=> nil

Explaining the terms in the parsed and compiled code is beyond the scope of this blog post, and I recommend reading Ruby Under a Microscope.

Newer versions of ruby also allows us to compile the code to binary and execute them later. it is experimental and platform dependent.

? cat example.rb
number = 23
puts number + 23
? ruby -e "File.write('example.bin',RubyVM::InstructionSequence.compile_file('example.rb')
.to_binary)"
? cat example.bin
YARB@
     ?x86_64-darwin18%?#?%?gw
numberE+Eexampleputs?????????%

irb(main):018:0>  RubyVM::InstructionSequence.load_from_binary(File.read('example.bin')).eval
46

---

That is it for the ruby compilation process for now. Now let’s how the ruby’s require method works.

Contrary to what I thought initially, require is not a keyword in ruby, it is a method from ruby’s Kernal module.

Let’s look at the overly simplified version of require method for our context.

def require(file_name)
  eval File.read(filename)
end

---

Two main issues with this implementation are

1. Requiring same file again will load the file again
2. Only absolute paths are supported

We can fix these issues in the following ways

  $LOADED_FEATURES = []
  def require(filename)
    return false if $LOADED_FEATURES.include?(filename)
    eval File.read(filename)
    $LOADED_FEATURES << filename
  end

  $LOAD_PATH = []
  #$LOAD_PATH = += gems_path + stdlib path + application code paths

  def require(filename)
    full_path = $LOAD_PATH.take do |path|
      File.exist?(File.join(path, filename))
    end

    eval File.read(full_path)
  end

While above code snippets are dummy implementations, Ruby actually uses the constants $LOADED_FEATURES and $LOAD_PATH for the same use case. Here is a stat from one of our app for reference

irb(main):054:0>  $LOADED_FEATURES.count
=> 6552

irb(main):058:0> $LOAD_PATH.count
=> 779

Another important method we need to recall is the fork system call in POSIX systems. fork allows the OS to create a new process as the child process with the same memory space. Modern hardware architectures like x86 allows the OS to optimise the fork with a mechanism called copy on write. In short it is very cheap to create a forked process of an app than loading that app from scratch.

Now let’s look at the gems from the title.

Spring is a rails only tool to speedup development and test environments. It creates your app process in the background for development and test environments and acts as a server. When you run a process like bundle exec rails server or bundle exec rspec it sends the data like command, ENV values, arguments etc to spring server, and spring server will fork the server process and run the task.

Since the server process already loaded the app, the loading time of the app will be negligible and your task will run faster.

But when the code changes, the server process needs to update and it may fail due to various reasons, like adding new directories. That is why you ended up having to run spring stop manually or restart the system to make the application behave as expected.

The serve method in Spring is provided as reference. You can see zipping io values, fetching arguments, env etc from client, forking the new process etc in the below snippet.

 def serve(client)
      log "got client"
      manager.puts

      _stdout, stderr, _stdin = streams = 3.times.map { client.recv_io }
      [STDOUT, STDERR, STDIN].zip(streams).each { |a, b| a.reopen(b) }

      preload unless preloaded?

      args, env = JSON.load(client.read(client.gets.to_i)).values_at("args", "env")
      command   = Spring.command(args.shift)

      connect_database
      setup command

      if Rails.application.reloaders.any?(&:updated?)
        Rails.application.reloader.reload!
      end

      pid = fork {
        Process.setsid
        IGNORE_SIGNALS.each { |sig| trap(sig, "DEFAULT") }
        trap("TERM", "DEFAULT")

        unless Spring.quiet
          STDERR.puts "Running via Spring preloader in process #{Process.pid}"

          if Rails.env.production?
            STDERR.puts "WARNING: Spring is running in production. To fix "         \
                        "this make sure the spring gem is only present "            \
                        "in `development` and `test` groups in your Gemfile "       \
                        "and make sure you always use "                             \
                        "`bundle install --without development test` in production"
          end
        end

        ARGV.replace(args)
        $0 = command.exec_name

        # Delete all env vars which are unchanged from before Spring started
        original_env.each { |k, v| ENV.delete k if ENV[k] == v }

        # Load in the current env vars, except those which *were* changed when Spring started
        env.each { |k, v| ENV[k] ||= v }

        # requiring is faster, so if config.cache_classes was true in
        # the environment's config file, then we can respect that from
        # here on as we no longer need constant reloading.
        if @original_cache_classes
          ActiveSupport::Dependencies.mechanism = :require
          Rails.application.config.cache_classes = true
        end

        connect_database
        srand

        invoke_after_fork_callbacks
        shush_backtraces

        command.call
      }

      disconnect_database

      log "forked #{pid}"
      manager.puts pid

      wait pid, streams, client
    rescue Exception => e
      log "exception: #{e}"
      manager.puts unless pid

      if streams && !e.is_a?(SystemExit)
        print_exception(stderr, e)
        streams.each(&:close)
      end

      client.puts(1) if pid
      client.close
    ensure
      # Redirect STDOUT and STDERR to prevent from keeping the original FDs
      # (i.e. to prevent `spring rake -T | grep db` from hanging forever),
      # even when exception is raised before forking (i.e. preloading).
      reset_streams
    end

Bootsnap is a gem released by shopify by extracting boot time improvements they made in their app. We can categorise them into two parts

Path prescanning

• Kernel#require and Kernel#load are modified to eliminate $LOAD_PATH scans
• ActiveSupport::Dependencies.{autoloadable_module?,load_missing_constant,depend_on} are overridden to eliminate scans of ActiveSupport::Dependencies.autoload_paths.

Compilation Caching

• RubyVM::InstructionSequence.load_iseq is implemented to cache the result of Ruby bytecode compilation
• YAML.load_file is modified to cache the result of loading a YAML object in MessagePack format (or Marshal, if the message uses types unsupported by MessagePack)

If you look at the pseudo code above to demonstrate the LOAD_PATH behaviour, you will see that we need to check file existence every time we do a require, which is an io operation and not very cheap to perform. What if we can do something like this?

 def require(filename)
    if $CACHED_PATH[file_name]
      full_path = $CACHED_PATH[filename]
    else
      full_path = $LOAD_PATH.take do |path|
        File.exist?(File.join(path, filename))
      end
    end

    eval File.read(full_path)
 end

load path for a library is not something that changes very often, especially for gem paths and standard library paths, bootsnap caches them to save redundant file checks. Not that cache duration and expiration vary with files, so caching them in a constant won’t work. CACHED_PATH is just for reference and not used by the gem.

Another important optimization by bootsnap is the compilation cache. We covered the ruby compilation process in the beginning and saw that every single file needed to go through the compilation process every single time they got called. Bootsnap addresses this by caching yarv code(instruction sequence) and compiling the code only if the code changes.

There are also other optimised YAML loading by caching in an optimised format. overall Bootsnap gives impressive benefits as it works on production as well. We got about 30% reduction in boottime for our production ecommerce app.

Note: This was originaly presented in internal tech talk in Sephora and made some tweaks to the content for wider audience.

]]> Aboobacker M K https://mrkaran.dev/posts/coredns-vector-clickhouse/ https://mrkaran.dev/posts/coredns-vector-clickhouse/ Fri, 04 Jun 2021 18:30:00 GMT [^]]+)]\s(?P[^:]+):(?P\S+)\s+-\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P[^"]+)"\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P[\d\.]+).* ERROR logs: \[(?PERROR)]\s+(?Pplugin\/errors):\s+(?P\S)+\s+(?P\S+)\s+(?P[^:]*):\s+(?P.*) Combining a bunch of other things to remove some fields and constructing the final payload, the config looks like this: # Parse coredns logs [transforms.parse_logs] type = "remap" inputs = ["coredns_logs"] source = ''' # parse the log event. ts = .timestamp log,err = parse_regex(.message,r'\[(?P[^]]+)]\s(?P[^:]+):(?P\S+)\s+-\s+(?P\S+)\s+"(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P[^"]+)"\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P[\d\.]+).*') if err !=null { # capture the error log. If the error log also fails to get parsed, the log event is dropped. log = parse_regex!(.message,r'\[(?PERROR)]\s+(?Pplugin/errors):\s+(?P\S)+\s+(?P\S+)\s+(?P[^:]*):\s+(?P.*)') } . = log # add timestamp .timestamp = ts # remove fields we dont care about del(.do) ''' drop_on_error = true Apart from regex matching, we store the timestamp as received from Vector (since CoreDNS logs don’t contain any timestamp information). We delete some fields that we don’t care about. Vector uses a powerful DSL (called VRL) to do such kind of transformations on the fly. It has a lot of functions to do almost any kind of transformation to your original event payload. You can invoke vector vrl from the terminal and get a shell to write the above transformations and debug quickly. It proved to be really useful when dealing with such a long regex pattern. Storing in Clickhouse# Finally we get to the part where we need to dump these logs to our Clickhouse DB. Here’s the schema for the table where we will be storing these records: CREATE DATABASE IF NOT EXISTS `coredns`; CREATE TABLE IF NOT EXISTS `coredns`.`logs` ( `timestamp` DateTime('Asia/Kolkata'), `bufsize` Int32, `class` LowCardinality(String), `duration` Float64, `id` Int32, `level` LowCardinality(String), `name` String, `proto` LowCardinality(String), `rcode` LowCardinality(String), `rflags` String, `server_addr` String, `server_port` Int32, `rsize` Int32, `size` Int32, `type` LowCardinality(String) ) ENGINE = MergeTree() PARTITION BY toYYYYMM(timestamp) ORDER BY toYYYYMMDD(timestamp) TTL timestamp + INTERVAL 1 WEEK; Key things to note: LowCardinality is used for columns where the data is predictable to reduce the disk space used. Clickhouse uses the sort key as the primary key if unspecified. This is the default behaviour. TTL for the records is set to 1 week. After 1 week all the records will be purged. Since this is my dev machine, I don’t really care about a higher TTL. This also means that the partition is never really put to use since I am partitioning by month but logs are being deleted every week. At this scale, it doesn’t really make sense to even have it, but I just included it for posterity. UPDATE: Clickhouse on Twitter clarified that ORDER BY timestamp will have better performance in this context. Usually if your queries are “last 1h”, “last 5m” based, it is better to not store the the sort key as YYYYMMDD format. Now, we need to instruct Vector to send these logs to Clickhouse: [sinks.coredns_logs_clickhouse_output] type = "clickhouse" inputs = ["parse_logs"] compression = "gzip" database = "coredns" endpoint = "http://localhost:8123" table = "logs" encoding.timestamp_format = "unix" batch.timeout_secs = 10 Clickhouse offers an HTTP API (which is running on port 8123 by default). Vector takes the input from the previous step (parse_logs transformation) and sends it to Clickhouse over the HTTP interface. Clickhouse stores datetimes in UNIX, so before sending the data, Vector can encode certain fields in the payload to a different data type as well (isn’t that cool 😎) Query Examples# I’ve been running this pipeline for 3-4 days, so I have a decent amount of data collected to show for the blog post. Total Count of Queries SELECT count(*) FROM coredns.logs ┌─count()─┐ │ 16774 │ └─────────┘ Top Query Types SELECT count(*) AS total, type FROM coredns.logs GROUP BY type ORDER BY total DESC LIMIT 5 ┌─total─┬─type─┐ │ 9931 │ A │ │ 6852 │ AAAA │ └───────┴──────┘ Top Query Names SELECT count(*) AS total, name FROM coredns.logs GROUP BY name ORDER BY total DESC LIMIT 5 ┌─total─┬─name────────────────────────────┐ │ 2513 │ ping.archlinux.org. │ │ 1868 │ ws.todoist.com. │ │ 1011 │ incoming.telemetry.mozilla.org. │ │ 802 │ vortex.data.microsoft.com. │ │ 707 │ logs-01.loggly.com. │ └───────┴─────────────────────────────────┘ Max/Min duration SELECT max(duration) AS max, min(duration) AS min FROM coredns.logs FORMAT Vertical Row 1: ────── max: 4.056606352 min: 0.000020837 Top TLDs Queried SELECT count(name) AS total, topLevelDomain(substring(name, 1, -1)) AS tld FROM coredns.logs GROUP BY tld ORDER BY total DESC LIMIT 10 ┌─total─┬─tld──┐ │ 10666 │ com │ │ 3950 │ org │ │ 671 │ net │ │ 346 │ so │ │ 288 │ tech │ │ 279 │ io │ │ 190 │ co │ │ 167 │ dev │ │ 82 │ arpa │ │ 43 │ in │ └───────┴──────┘ Well, that’s all I could think of really. If you’ve some more interesting analysis to get from this data, let me know! Summary# The intention behind writing this post is to give an overview of how the entire log collection and processing pipeline works. Using Vector has been an amazing experience however the sad bit is that I don’t know Rust and I cannot contribute to some of the issues I’ve opened (even though they are presumably trivial). Maybe I should pick up Rust, finally? 🤭 Thanks for reading!]]> I’ve been toying around Clickhouse and Vector at my day job and find both of these tools pretty interesting at what they do. A short summary for those unaware of these tools:

• Vector helps you build a pipeline for collecting, transforming and processing different kinds of observability data (logs and metrics).
• Clickhouse is a columnar based DB used as a warehousing tool for generating reports and analytics.

Now, for the context, I use coredns on my personal dev machine as it supports Split DNS (routing certain zones to a particular resolver) which I need for accessing internal domains at work. Yep, systemd-resolved can also do this, but I find coredns easier to configure and manage with OpenVPN as well.

Anyway, so one random evening, I got the idea of dumping CoreDNS Logs to Clickhouse. Maaaybe I was still hungover from the Vector/Clickhouse work I was doing at work but nevertheless I was interested in it.

Overview#

• Gather logs from CoreDNS.
• Transform the logs with regex and construct a payload for Clickhouse.
• Write a schema for the logs table.
• Dump to Clickhouse.

Here’s how the pipeline looks:

Collecting Logs#

First, let’s look at how the raw logs are structured by CoreDNS:

[INFO] 127.0.0.1:55678 - 21963 "A IN xpui.app.spotify.com. udp 38 false 512" NXDOMAIN qr,rd,ra 121 0.061416978s
[INFO] 127.0.0.1:59333 - 22742 "AAAA IN incoming.telemetry.mozilla.org. udp 48 false 512" NOERROR qr,aa,rd,ra 106 0.049235139s
[INFO] 127.0.0.1:39609 - 47247 "AAAA IN ping.archlinux.org. udp 36 false 512" NOERROR qr,rd,ra 140 0.056721154s

Vector provides a variety of sources to collect these logs. Since I am running coredns as a Docker container, the following config shows how to collect logs from a particular container:

# Filter coredns logs from Docker logs
[sources.coredns_logs]
  type = "docker_logs" # required
  docker_host = "unix:///var/run/docker.sock"
  include_images = ["coredns/coredns"] # optional, no default

The above config basically specifies a Docker Host variable and an image name filter. Vector talks to the Docker API over a unix socket and gathers metadata about the container (like container_created_at, container_name, label etc).

After collecting logs from Vector, it enriches with the following metadata.

{
    "container_created_at": "2021-06-04T14:18:03.967143133Z",
    "container_id": "00c5c4d36ea5b4772b517d3cca7d397c92f72be2a2bf45bb8c430f717fbd331e",
    "container_name": "coredns_coredns_1",
    "host": "iris",
    "image": "coredns/coredns",
    "label": {
        "com.docker.compose.config-hash": "928d71143c2af6553d551dbbf14140304d53f92378746454fbfeb0382a896d5b",
        "com.docker.compose.container-number": "1",
        "com.docker.compose.oneoff": "False",
        "com.docker.compose.project": "coredns",
        "com.docker.compose.project.config_files": "/home/karan/Code/Infra/coredns/hydra-vpn-compose.yml",
        "com.docker.compose.project.working_dir": "/home/karan/Code/Infra/coredns",
        "com.docker.compose.service": "coredns",
        "com.docker.compose.version": "1.29.2"
    },
    "message": "[INFO] 127.0.0.1:38266 - 20341 \"A IN open.spotify.com. udp 34 false 512\" NOERROR qr,rd,ra 160 0.300268123s",
    "source_type": "docker",
    "stream": "stdout",
    "timestamp": "2021-06-04T16:13:07.454601872Z"
}

(NOTE: I am using console sink to dump these logs to STDOUT. It’s pretty handy for inspecting logs).

# Print parsed logs to stdout
[sinks.print]
type = "console"
inputs = ["coredns_logs"]
encoding.codec = "json"

As you can see from the above JSON object, Vector has transformed the log with its own Data Model. The log that we care about now is inside .message key. It’s nice to have other metadata as well.

Transforming the logs#

Our objectives at this step:

• Discard unused fields. We don’t really care about container metadata for this mini-project.
• Parse the message field with regex so they can be stored in individual columns in our Clickhouse table.

Now, CoreDNS can emit two kinds of logs (INFO and ERROR). The error usually happens when the upstream resolver is unreachable or there’s an issue with any of the CoreDNS plugins.

We need to write regex for both cases:

1. INFO logs:

\[(?P<level>[^]]+)]\s(?P<server_addr>[^:]+):(?P<server_port>\S+)\s+-\s+(?P<id>\S+)\s+(?P<type>\S+)\s+(?P<class>\S+)\s+(?P<name>\S+)\s+(?P<proto>\S+)\s+(?P<size>\S+)\s+(?P<do>\S+)\s+(?P<bufsize>[^"]+)"\s+(?P<rcode>\S+)\s+(?P<rflags>\S+)\s+(?P<rsize>\S+)\s+(?P<duration>[\d\.]+).*

2. ERROR logs:

\[(?P<level>ERROR)]\s+(?P<component>plugin\/errors):\s+(?P<code>\S)+\s+(?P<name>\S+)\s+(?P<type>[^:]*):\s+(?P<error_msg>.*)

Combining a bunch of other things to remove some fields and constructing the final payload, the config looks like this:

# Parse coredns logs
[transforms.parse_logs]
type = "remap"
inputs = ["coredns_logs"]
source = '''
# parse the log event.
ts = .timestamp
log,err = parse_regex(.message,r'\[(?P<level>[^]]+)]\s(?P<server_addr>[^:]+):(?P<server_port>\S+)\s+-\s+(?P<id>\S+)\s+"(?P<type>\S+)\s+(?P<class>\S+)\s+(?P<name>\S+)\s+(?P<proto>\S+)\s+(?P<size>\S+)\s+(?P<do>\S+)\s+(?P<bufsize>[^"]+)"\s+(?P<rcode>\S+)\s+(?P<rflags>\S+)\s+(?P<rsize>\S+)\s+(?P<duration>[\d\.]+).*')
if err !=null {
    # capture the error log. If the error log also fails to get parsed, the log event is dropped.
  log = parse_regex!(.message,r'\[(?P<level>ERROR)]\s+(?P<component>plugin/errors):\s+(?P<code>\S)+\s+(?P<name>\S+)\s+(?P<type>[^:]*):\s+(?P<error_msg>.*)')
}
. = log
# add timestamp
.timestamp = ts
# remove fields we dont care about
del(.do)
'''
drop_on_error = true

Apart from regex matching, we store the timestamp as received from Vector (since CoreDNS logs don’t contain any timestamp information). We delete some fields that we don’t care about.

Vector uses a powerful DSL (called VRL) to do such kind of transformations on the fly. It has a lot of functions to do almost any kind of transformation to your original event payload. You can invoke vector vrl from the terminal and get a shell to write the above transformations and debug quickly. It proved to be really useful when dealing with such a long regex pattern.

Storing in Clickhouse#

Finally we get to the part where we need to dump these logs to our Clickhouse DB. Here’s the schema for the table where we will be storing these records:

CREATE DATABASE IF NOT EXISTS `coredns`;

CREATE TABLE IF NOT EXISTS `coredns`.`logs` (
    `timestamp` DateTime('Asia/Kolkata'),
    `bufsize` Int32,
    `class` LowCardinality(String),
    `duration` Float64,
    `id` Int32,
    `level` LowCardinality(String),
    `name` String,
    `proto` LowCardinality(String),
    `rcode` LowCardinality(String),
    `rflags` String,
    `server_addr` String,
    `server_port` Int32,
    `rsize` Int32,
    `size`  Int32,
    `type` LowCardinality(String)
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(timestamp)
ORDER BY toYYYYMMDD(timestamp)
TTL timestamp + INTERVAL 1 WEEK;

Key things to note:

• LowCardinality is used for columns where the data is predictable to reduce the disk space used.
• Clickhouse uses the sort key as the primary key if unspecified. This is the default behaviour.
• TTL for the records is set to 1 week. After 1 week all the records will be purged. Since this is my dev machine, I don’t really care about a higher TTL.
• This also means that the partition is never really put to use since I am partitioning by month but logs are being deleted every week. At this scale, it doesn’t really make sense to even have it, but I just included it for posterity.

UPDATE:

Clickhouse on Twitter clarified that ORDER BY timestamp will have better performance in this context. Usually if your queries are “last 1h”, “last 5m” based, it is better to not store the the sort key as YYYYMMDD format.

Now, we need to instruct Vector to send these logs to Clickhouse:

[sinks.coredns_logs_clickhouse_output]
  type = "clickhouse"
  inputs = ["parse_logs"]
  compression = "gzip"
  database = "coredns"
  endpoint = "http://localhost:8123"
  table = "logs"
  encoding.timestamp_format = "unix"
  batch.timeout_secs = 10

Clickhouse offers an HTTP API (which is running on port 8123 by default). Vector takes the input from the previous step (parse_logs transformation) and sends it to Clickhouse over the HTTP interface. Clickhouse stores datetimes in UNIX, so before sending the data, Vector can encode certain fields in the payload to a different data type as well (isn’t that cool 😎)

Query Examples#

I’ve been running this pipeline for 3-4 days, so I have a decent amount of data collected to show for the blog post.

• Total Count of Queries

  SELECT count(*)
  FROM coredns.logs
  
  ┌─count()─┐
  │   16774 │
  └─────────┘

• Top Query Types

  SELECT
      count(*) AS total,
      type
  FROM coredns.logs
  GROUP BY type
  ORDER BY total DESC
  LIMIT 5
  
  ┌─total─┬─type─┐
  │  9931 │ A    │
  │  6852 │ AAAA │
  └───────┴──────┘

• Top Query Names

  SELECT
      count(*) AS total,
      name
  FROM coredns.logs
  GROUP BY name
  ORDER BY total DESC
  LIMIT 5
  
  ┌─total─┬─name────────────────────────────┐
  │  2513 │ ping.archlinux.org.             │
  │  1868 │ ws.todoist.com.                 │
  │  1011 │ incoming.telemetry.mozilla.org. │
  │   802 │ vortex.data.microsoft.com.      │
  │   707 │ logs-01.loggly.com.             │
  └───────┴─────────────────────────────────┘

• Max/Min duration

  SELECT
      max(duration) AS max,
      min(duration) AS min
  FROM coredns.logs
  FORMAT Vertical
  
  Row 1:
  ──────
  max: 4.056606352
  min: 0.000020837

• Top TLDs Queried

  SELECT
      count(name) AS total,
      topLevelDomain(substring(name, 1, -1)) AS tld
  FROM coredns.logs
  GROUP BY tld
  ORDER BY total DESC
  LIMIT 10
  
  ┌─total─┬─tld──┐
  │ 10666 │ com  │
  │  3950 │ org  │
  │   671 │ net  │
  │   346 │ so   │
  │   288 │ tech │
  │   279 │ io   │
  │   190 │ co   │
  │   167 │ dev  │
  │    82 │ arpa │
  │    43 │ in   │
  └───────┴──────┘

Well, that’s all I could think of really. If you’ve some more interesting analysis to get from this data, let me know!

Summary#

The intention behind writing this post is to give an overview of how the entire log collection and processing pipeline works. Using Vector has been an amazing experience however the sad bit is that I don’t know Rust and I cannot contribute to some of the issues I’ve opened (even though they are presumably trivial). Maybe I should pick up Rust, finally? 🤭

Thanks for reading!

]]> Karan Sharma https://captnemo.in/blog/2021/05/14/amazon-website-order-drm/ https://captnemo.in/blog/2021/05/14/amazon-website-order-drm/ Fri, 14 May 2021 00:00:00 GMT The Amazon US website allows you to export your Order History easily by visiting the “Order History Reports” page. No such option seems to exist for the Amazon websites for other countries. I was trying to write a simple scraper for the Amazon India Order History Page to get the same data, and discovered something interesting: Amazon encrypts the Order history page, and decrypts it using client side cryptography¹. If you were to visit the page, and check the response HTML, you’d see something like this in the source code (fairly simplified):

// Define encrypted content in JS
var payload = {
  "kid": "b70014",
  "iv": "/HenfXwYrGrrw8ff",
  "ct": "Wt78pPcibe8HAdVtoJ8+E9EGwt4IQYNghBMubBy7Zy/..."
}
// The HTML div to be populated with the decrypted HTML
var elementId = "csd-encrypted-889C1D02..";
// if client side decryption library failed to load
if (!window.SiegeClientSideDecryption) {
  window.location.href = "?disableCsd=missing-library";
  return;
}
// Decrypt and populate the div
SiegeClientSideDecryption.decryptInElementWithId(
  elementId, payload, {callSource: "now"}
);

The easiest way to scrape with such hurdles is often to just run a complete browser to scrape the site. The browser runs the javascript code with the decryption routine so you can scrape the actual content. However, it is much slower, and wastes CPU cycles - I try to avoid it if I can.

I could have spent time to parse the encryption routine, extract the key and decrypt the payload. But I found a much simpler solution - Amazon offers an alternate URL which disables encryption. As a fallback, in case the decryption code fails, it adds a query parameter ?disableCsd=missing-library. That disables the server side encryption entirely.

So if you’re trying to scrape Amazon and stumped at the missing order history in the HTML, try visiting the following URLs instead:

• https://www.amazon.in/gp/css/order-history?disableCsd=missing-library
• https://www.amazon.co.uk/gp/css/order-history?disableCsd=missing-library
• https://www.amazon.com/gp/css/order-history?disableCsd=missing-library

Amazon also sets a cookie csd-key=disabled but I didn’t experiment with that much.

Request My Data

Another alternative to scraping is to request Amazon for your data. Check the Retail.OrderHistory CSV files in the data export. The export from amazon.com includes data for other countries as well. The feature is also available on other Amazon sites:

• Amazon US - Request My Data
• Amazon India - Request My Data
• Amazon UK - Request My Data
• Amazon Germany - Request My Data

1. I’m hesitant to call this DRM, but it might qualify as such. ↩

]]> Nemo https://aboobacker.in/2021/04/30/intermediate-postgresql-for-rails-developers-part-0-get-your-enviornoments-ready.html https://aboobacker.in/2021/04/30/intermediate-postgresql-for-rails-developers-part-0-get-your-enviornoments-ready.html Fri, 30 Apr 2021 17:42:00 GMT Postgresql is one of the most advanced open source database available in the market. Adherence to the SQL standards and super cool extra features are few of the reason by Postgres being the most popular database among rails community. This is neither introduction to Postgres nor rails. This series is for the folks who have been using rails and Postgres for one or more years.

In part 0 we will walk through the tips for postgres setup to increase the productivity.

ActiveRecord orm which is built into rails provides a lot of cool methods to access the data in rubyistic way. But it is important to get familiarize with the native Postgres tooling for many tasks.

Psql

Psql is the official commandline client for postgresql, rails does have shortcut to open psql using the configuration from database configuration as rails dbconsole. But you can also open by using psql command on your command line client.

psql -h localhost -U username databasename

Above is the basic syntax for opening psql console, Once you enter that you will be greeted with psql console if the credentials given are correct.

Now let’s go through few psql tips

Backward slash commands

psql supports a lot of useful configuration options to improve the experiance in postgresql shell.

• \timing

  Timing command add the time taken to run the command, this is handy when optimizing query performance

[local] abomk@cv_dump=#SELECT COUNT(id) from users;
? 45129 ?
?????????

[local] abomk@cv_dump=#\timing
Timing is on.
[local] abomk@cv_dump=#SELECT COUNT(id) from users;
? 45129 ?
?????????

Time: 8.412 ms

• \s

  Get posgtres commands history, this is handy for documenting the stuff after doing experiments with different queries and many more. You can also choose to save the history by providing filename as the parameter as \s filename

• \i filename

Load query from file. Big queries are often convent to write in a text editor, \i enables loading a query from a file and execute it.

• \e

This another approach to solve the difficulty of writing multi line queries in a shell environment, When you give \e command, psql will open the text editor you set $EDITOR to open the query. You can edit and close the file, content will be copied to posql shell, and you can execute.

You can find more such commands from posgrestutorials and pgdash

Managing multiple versions of postgres

It is important to have the same version of postgres in your development setup as the production version. If you have multiple apps with diffrent versions of postgres versions, you can use pgenv to configure multiple versions of Postgres in your local. There are other options like asdf-postgres

Use different pager

The default pager is somewhat difficult to read when there are too many columns. pspg is an alternate pager you can use with Postgres

Save your psql settings in your psqlrc

You can save the psql configurations we discussed above in a file so that you don’t need to repeat every time you open the psql shell. Here is my psql configuration for reference

\set ON_ERROR_ROLLBACK interactive
\set COMP_KEYWORD_CASE upper
\set HISTFILE ~/.psql/history- :DBNAME
\set VERBOSITY verbose
\set PROMPT1 '%[%033[1m%]%M %n@%/%R%[%033[0m%]%#'
\setenv PAGER pspg
\pset border 2
\pset linestyle unicode
\set null '(null)'

You can find explanation of individual configurations from thoughtbot blog

Pgcli

Pgcli is an alternate postgresql client with additional features like autocompletion of table and sql queries.

We will go through more tips in coming articles, while it is not mandatory to follow the above setup for cominig articles, it is good to have to get hands on with psql shell

]]> Aboobacker M K https://mrkaran.dev/posts/how-i-take-notes/ https://mrkaran.dev/posts/how-i-take-notes/ Wed, 14 Apr 2021 02:40:55 GMT Over the past 2-3 years, note-taking apps have become all the rage. Note-taking is an extremely subjective topic and a lot of it depends on the individual’s workflow. There’s no one size fits all and maybe that justifies the ever-expanding landscape of such apps. I’ve tried a few popular ones (Notion, Roam Research) in the recent past but never really quite stuck to any after the initial hype phase.

I even collaborated with @iamd3vil to make our own version of the Zettelkasten based note-taking app. I found the Zettelkasten system to be really useful on paper but then again, I didn’t use it after a few weeks.

Tried the old school way of Bullet Journal (and I did end up liking it quite a lot) but it was not so practical in many cases (like documenting code snippets, URLs etc).

Disgruntled with all the options, I just had a simple folder on my laptop with some markdown files in it. It was a stop-gap solution until I found something better.

TL;DR: I’ve been through a pendulum phase of finding a new note-taking app, wasting time to set it up the “proper way” and then just end up not using it.

Enter Joplin#

Thanks to @shantanugoel who introduced me to Joplin. He’s a heavy user of it as well and that gave me some confidence to try it out. I initially disliked it because of not-so-great-looking UI theme and how it essentially looked just an editor. Admittedly, I was proven wrong quickly in my initial judgement. As I gave more time to it, I noticed I kept coming back to Joplin “naturally” and stuck through it because it’s so damn simple to use. Notion, for people who have tried it would know it complicates a lot of simple tasks. You need to create databases to render a simple table, every component is a “block” (a new page) and yes it’s slow (although they are working on it to make it better, just to be fair). Notion focuses a lot on team collaboration features, which I didn’t need for my “personal” note-taking system.

TL;DR: Joplin is fast, it’s open-source, it’s based on Markdown, and it’s simple to use. A tool you just forget that it exists, because it becomes a natural extension to your workflow. It has a great plugin system that you can use to extend it and build your own utilities on top of it. The search is based on sqlite3 FTS which is pretty awesome!

Workflow#

Joplin revolves around the concept of notebooks. Notebooks are broader categories for your content and you can nest multiple subnotebooks for specific categories.

I’ve the following notebooks and subnotebooks in my Joplin setup:

- Bookmarks
  - Twitter Threads
  - HN Threads
  - Articles
  - Design Inspo
  - Youtube Videos
- Inbox
  - Links
  - Adhoc Notes
- Personal
  - Finance
  - Dev Setup
  - OSS Ideas
  - Self Hosted Setup
- Work
  - Org-Stuff
  - Redacted

Over the week, I primarily use the Inbox/Adhoc Notes notebook as a brain dump. I don’t focus much on the structure, the aim is to get the content out and stored. I’m also someone who doesn’t like to keep more than 5 browser tabs open at any time, so I use Links notebook with Joplin’s Web Clipper service to store these links to read later.

Every weekend, I clean up these notebooks to achieve “Inbox Zero”. The idea is to move all these ad-hoc notes to their proper notebooks, annotated with tags. All the useful links are moved in the appropriate Bookmarks/... notebook as well. This helps me find stuff quicker at a later time.

I heavily use Tags in all my notebooks, which allows me to have a unified view of different kind of stuff I’ve. For example “golang” tag in my Work notes and Personal notes, allows me to see all the “golang” stuff together in one place.

For stuff that can be shared publicly, I basically copy-paste those notes in my public wiki as well. This allows me to share snippets/commands with others, which Joplin cannot do.

Synchronisation#

Joplin provides a bunch of different sync options. I’ve tried Dropbox, Nextcloud and AWS S3 targets in the past but off late there’s a new sync option, Joplin Server which provides native sync for Joplin files. I found this option to be the best so far because Dropbox/OneNote etc have API limits and syncing on an initial device with lots of notes will be time-consuming.

I self-host this Joplin Sync Server on my server and have configured the Android app and the Desktop app to use this server endpoint as the sync target. So far so good, although it’s relatively a newer sync option so it’s pertinent to have alternate backups.

Backups and Export#

Joplin stores all the files locally on a device in a sqlite3 DB. It can export notes in markdown/HTML format however the file names are all named with the id of the note (and not really the title of the note). I found this to be a bit of a drawback, however, one can quickly whip up a small Python script to fix this using the Joplin API.

Joplin also has the option to export all of the notes and notebooks with their metadata (Geolocation, creation time etc), tags in a custom format called Joplin Export File (JEX). This option is pretty convenient to re-import in a new Joplin installation as well.

Sidenote for people using Joplin Server: Once https://github.com/laurent22/joplin/issues/4836 gets resolved, it’ll be possible to do joplin sync and cron it, just like other sync targets.

Support#

Joplin is 100% FOSS and is actively developed by @laurent22 and a few other regular contributors. I contribute $5/mo to laurent22 via Github Sponsors. It’s more or less the same amount that most note-taking apps charge for personal use as well, so this is just me expressing gratitude for building such a lovely software for the world to use it. I’m not sure of the motivations of laurent22 behind building this and I don’t wanna incorrectly assume anything either, but I guess some amount of financial incentive makes the whole deal sustainable for the open-source ecosystem.

Thanks for reading! It’s been around a year that I am using Joplin and I posted this blog post only after really really using it a lot.

I’d love to know about your note-taking setups too, so please reach out to me on the usual channels that I’m available on and feel free to discuss!

Fin!

(Bonus Section) Why Not Obsidian#

Yes, Obsidian is comparable to Joplin in a lot of ways. However there’s a term in the license of Obsidian for personal use that makes it impossible to use it for your work stuff:

You need to pay for Obsidian if and only if you use it for revenue-generating, work-related activities in a company that has two or more people. Get a commercial license for each user if that’s the case

I don’t have a problem for “paying” for software but such kinda licenses are just BS.

]]> Karan Sharma https://aboobacker.in/2021/03/28/intellectual-property-open-source-and-mimemagic.html https://aboobacker.in/2021/03/28/intellectual-property-open-source-and-mimemagic.html Sun, 28 Mar 2021 18:47:00 GMT The alleged GPL violation in a ruby library called mimemagic which broke builds of rails projects across the world brought discussion of Intellectual property in the software world one more time last week. It is important to have a general idea about these terms for a software developer to avoid getting into big troubles in the future.

This is my attempt to write down my understanding about open source and Intellectual property, I am not a lawyer and the implementation of IP is based on the local laws which is not universal.

Though the modern concept of IP is originated from the exclusive access to rights and grants by the British monarch, it soon became a mechanism to promote the creation of intellectual goods, by giving the inventors some rights to the work they made.

Though IP may sound an alien term, it is something critical for any company, for instance, company name, logo, design of the logo etc are trademarks which is a kind of intellectual property. This enables companies to protect reputation and goodwill of the brand by preventing others from copying same brand symbols to sell their products.

Content produced by the company is protected by copyright, this also includes the software they made, in almost all legislations content produced is inherently owned by the author and protected by copyright, this prevents unauthorized access and redistribution of content by other people.

Another important intellectual property is patent which provides protection for innovative ideas, unlike copyright patent requires registration within a particular timeframe to the government agency. In general Patent protects ideas and copyright protects manifestation of that idea

There are other IPs like trade secrets which I am not covering here to limit the scope of the article.

Historically hardware was the most expensive piece of a computer and program or the software that runs on them wasn’t valued as important. But as more software distribution patterns emerged people started distributing just binaries of the software which prevented users from changing the software as per their needs. Richard R Stallman who was working in MIT lab during this period had a big trouble because of this, they used to adjust the printer firmware as per their need of paper size, But the new printer firmware came with just binary format which prevented from making the modification which caused wastage of papers they already had, Stallman requested the firmware owner to share the source code, but he refused.

That incident hit him big as usage of a computer owned by the user is dictated by the firmware owner. He believed this is a big ethical and social issue. He launched the GNU project in 1983 to create an operating system consisting of only free software. As per his own words the word free of “free” as in “free speech,” not as in “free beer”. Many including Linux followed this approach and many free software programs became available in the market.

In late 90s some people in industry realized the advantage of having free software in the enterprise world, i.e. when the code is shared more people can learn and adapt from it and there by saving time and cost. They also a coined a term called “Open Source” since people may easily interpret free software as free of cost software.

While free softwares and open source softwares and practically largely the same, they are two movements with different motivations but almost the same path to achieve their goals. While free software movement is from the perspective of the user to have full control of his/her system, open source is from the perspective of software engineer or company to get benefit from shared source code.

Open source Initiative, which formalized the open source with 10 rules and listed software licenses compatible with these rules as open source licenses.

But why does an open source software need license?

The Answer is simple, as we discussed in the initial section, any content produced is owned by the author and others can’t redistribute without author’s permission. Licenses are contract from a software developer to the user to share some of his/her/their rights to the user, In the case of open source it will be freedom to redistribute, make derived work etc. In other words open source licenses Grants certain IP rights to users who are subject to license. This is different from EULA(End user license agreement) we see in the proprietary softwares which is to limit what the user can do with the software we purchased.

Also keep in mind that open source is not public domain which are the work which is not owned by anyone

There are many kinds of open source licenses ranging from permissive license which doesn’t ask the users anything more than attribution to copyleft licenses which mandate you to make your software of also open source if you use one in your software.

I’ll be writing another article about commonly used licenses and things to be taken care of while using such licenses in another post, till then you can refer this awesome website.

Now let’s go back to the incident that happened in the last week. freedesktop is a project on interoperability of free software projects, they maintain the list of mime types and is licensed under GPL. GPL is a very restrictive license and if you want to link your software to a GPLed software and distribute, then you will have to license your software with the same restrictions. In other words, you can’t directly link a software linked with a gpled software/code and distribute under proprietary or permissive open source license like MIT. The mimemagic gem which is MIT licensed used this GPLed code from freedesktop which is violation of GPL license. Initial fix to convert mimemagic license to GPL. But due to the nature of copyleft licenses, software which includes mimemagic will also have to comply GPL rules. So they later removed the GPLed file and made the library to use mime data from the user’s computer.

This may raise some questions about using GPL programs you use with your closed source(projects which are not open source) projects. First, you don’t have to care much about GPL in backend of software as service model like most websites does, as you are just distributing the output of the software, not the software itself to the end user, this is also called as saas loophole. Even if you aren’t, it is unlikely that mysql being part of your source code, if you are just mysql as database, you are using its SQL interface and not as part of your programs.

Now let’s look at the common errors developers make due to lack of awareness about IP,

• Copy and paste snippets from stackoverflow (read more here).

• Copy open source code snippets from GitHub

  Even if the code is open source, some may require making your code also open source, some require attribution.

• Copy code snippets from GitHub or other publically available places

  If you found some code on GitHub or any publically accessible places, you shouldn’t assume that code is open source unless license is explicitly given. For instance projects without explicit license in GitHub, all you can do is read the code and fork, you aren’t supposed to change the code

• Forking open source projects without changing trademarks

  Even when the open source project is licensed under permissive open source license, it doesn’t grant you to use it’s trademark for redistribution. For example you can’t simply fork mysql and sell it as mysql

• Open sourcing stuff without proper planning

  When you are open sourcing certain part of codebase, you must have a plan for what to do with the contributions, For instance when you release a component of your code base as open source and someone from community send a pull request and you merged to your repo, by default the person who contributed will have the copyright of the code, company can only use that code as per the license given in the repository. CLA Can be used as a workaround for this

Do you have more examples? Feel free to comment below.

We covered the general IP concepts and it’s significanse in open source, I am not a lawyer and this blogpost is not a legal opinion. So talk to your lawyer before making legal decisons

]]> Aboobacker M K https://mrkaran.dev/posts/home-server-nomad/ https://mrkaran.dev/posts/home-server-nomad/ Sun, 14 Feb 2021 02:40:55 GMT 1 in less than 30 minutes. This BTW is a strong sign of how easy Nomad is to get started with as compared to K8s. The sheer amount of different concepts you’ve to register in your mind before you can even deploy a single container in a K8s cluster is bizarre. Nomad takes the easy way out here and simplified the concepts for developers into just three things: job \_ group \_ task Job: Job is a collection of different groups. Job is where the constraints for type of scheduler, update strategies and ACL is placed. Group: Group is a collection of different tasks. A group is always executed on the same Nomad client node. You’ll want to use Groups for use-cases like a logging sidecar, reverse proxies etc. Task: Atomic unit of work. A task in Nomad can be running a container/binary/Java VM etc, defining the mount points, env variables, ports to be exposed etc. If you’re coming from K8s you can think of Task as a Pod and Group as a Replicaset. There’s no equivalent to Job in K8s. BUT! The coolest part? You don’t have to familiarise yourself with all different types of Replicasets (Deployments, Daemonsets, Statefulsets) and different ways of configuring them. Want to make a normal job as a periodic job in Nomad? Simply add the following block to your existing Job: periodic { cron = "@daily" } You want to make a service run as a batch job (on all Nomad nodes – the equivalent of Daemonset in K8s)? Simply make the following change to your existing job: -type="service" +type="batch" You see this is what I mean by the focus on UX. There are many many such examples which will leave a nice smile on your face if you’re coming from K8s background. I’d recommend reading Internal Architecture of Nomad if you want to understand this in-depth. Architecture# Tech stack for Hydra: Tailscale VPN: Serves as a mesh layer between my laptop/mobile and DO server. Useful for exposing internal services. Caddy for reverse proxying and automatic SSL setup for all services. I run 2 instances of Caddy: Internal: Listens on Tailscale Network Interface. Reverse proxies all private services. Public: Listens on DO’s Public IPv4 network interface. Reverse proxies all public-facing services. Terraform: Primary component to have IaC (Infra as Code). Modules to manage: Cloudflare DNS Zone and Records DO Droplet, Firewall rules, SSH Keys, Floating IPs etc. Nomad Jobs. Used for running workloads after templating env variables, config files in Nomad job files. Complexity of Nomad vs Kubernetes# Nomad shines because it follows the UNIX philosophy of “Make each program do one thing well”. To put simply, Nomad is just a workload orchestrator. It only is concerned about things like Bin Packing, scheduling decisions. If you’re running heterogeneous workloads, running a server (or a set of servers) quickly becomes expensive. Hence orchestrators tend to make sense in this context. They tend to save costs by making it efficient to run a vast variety of workloads. This is all an orchestrator has to do really. Nomad doesn’t interfere in your DNS setup, Service Discovery, secrets management mechanisms and pretty much anything else. If you read some of the posts at Kubernetes Failure Stories, the most common reason for outages is Networking (DNS, ndots etc). A lot of marketing around K8s never talks about these things. I always maintain “Day 0 is easy, Day N is the real test of your skills”. Anyone can deploy a workload to a K8s cluster, it’s always the Day N operations which involve debugging networking drops, mysterious container restarts, proper resource allocations and other such complex issues that require real skills and effort. It’s not as easy as kubectl apply -f and my primary gripe is with people who miss out on this in their “marketing” pitches (obvious!). When to use Nomad# Nomad hits the sweet spot of being operationally easy and functional. Nomad is a great choice if you want to: Run not just containers but other forms of workloads. Increase developer productivity by making it easier to deploy/onboard new services. Consistent experience of deployment by testing the deployments locally. (Not joking) You are tired of running Helm charts or writing large YAML manifests. The config syntax for Nomad jobs is human friendly and easy to grasp. Nomad is available as a single binary. If you want to try it locally, all you need is sudo nomad agent -dev and you’ll have a Nomad Server, Client running in dev mode along with a UI. This makes it easy for the developers to test out the deployments locally because there’s very little configuration difference between this and production deployment. Not to forget it’s super easy to self-host Nomad clusters. I’m yet to meet anyone who self hosts K8s clusters in production without a dedicated team babysitting it always. Once you eliminate the “blackbox” components from your stack, life becomes easier for everyone. When to not use Nomad# If you’re relying on custom controllers and operators. Operator Pattern is a new way of managing large complex distributed systems (like databases, job queues etc). There are a lot of community built operators which help in reducing the effort to run these services. However, all of these are tied deeply into the “Kubernetes” ecosystem. If you find yourself running any of such operators, it’ll be tough (not impossible) to translate the same in Nomad ecosystem. I genuinely cannot think of any other reason to not use Nomad! Practical Scenarios# Since I migrated a couple of workloads from my DO docker containers setup to Nomad, I’d demonstrate a few use cases which might be helpful if you want to start migrating your services to Nomad Accessing a Web service with Reverse Proxy# Context: I’m running Caddy as a reverse proxy for all the services. Since we discussed earlier, Nomad only is concerned about scheduling, so how exactly do you do Service Discovery? You need Consul (or something like Consul, Nomad has no hard restrictions) to register a service name with it’s IP Address. Here’s how you can do that: In the .task section of your Nomad job spec, you need to register the service name with the port you’re registering and additional tags as metadata (optional): service { name = "gitea-web" tags = ["gitea", "web"] port = "http" } Nomad’s template uses consul-template behind the scenes. This is a small utility which continuously watches for Consul/Vault keys and provides the ability to reload/restart your workloads if any of those keys change. It can also be used to discover the address of the service registered in Consul. So here’s an example of Caddyfile using Consul Template functions to pull the IP address of the upstream gitea-web service: git.mrkaran.dev { {{ range service "gitea-web" }} reverse_proxy {{ .Address }}:{{ .Port }} {{ end }} } When a job is submitted to Nomad, a rendered template is mounted inside the container. You can define actions on what to do when the values change. For eg on a redeployment of Gitea container, the address will most likely change. We’d like Caddy to automatically restart with the new address configured in the Caddyfile in that case: template { data = < It’s been a long time since I’ve written a post on Hydra (my home server). I use Hydra as a testbed to learn new tools, workflows and it just gives me joy to self-host applications while learning something in return.

History#

A brief history of how Hydra’s setup evolved over time:

2019:

• A pretty minimal K3s setup deployed on 2 RPi4 nodes. I couldn’t continue with this setup because:
  • Some of the apps didn’t have ARM-based image (this was 2019, pre M1 hype era).
  • Didn’t want to risk deploying persistent workloads on RPi.
  • A lot of tooling to deploy workloads was missing (storing env variables for eg.).
  • It was so boring to write YAML (that I also did at work). Didn’t give me joy.

2020 First Half:

• RPi 2x Nodes + K3s + DO Droplet. Tailscale for networking.
  • This was a considerable step up from the previous setup. I deployed a DO node and added Node Labels to deploy persistent workloads on DO Node only.
  • I used my own tooling Kubekutr + Kustomize which helped with version control of my configs.
  • Took quite a bit of time to onboard new services. Got lazy, didn’t host much apart from initial 3-4 applications.
  • Writing long YAMLs. No joy.

2020 Second Half:

• Single node on DO. Terraform for deploying Docker containers.
  • I believe the third iteration nailed it for me. I kept the setup super simple, used Terraform for deploying workloads as Docker containers.
  • Used Terraform extensively for setting up the node, Cloudflare records, DO firewall rules.
  • Time to onboard new services reduced from a couple of hours to a few minutes. This was a huge win for me. I deployed around 10-15 new services to try it out on the server directly.
  • Writing HCL is actually a much better experience than YAML.

Why Nomad#

Around a month back, Kailash had asked about feedback on Nomad. We at Zerodha (India’s largest stock broker) are evaluating it to migrate our services to Nomad from Kubernetes (more on this later). It was almost 2 years since I last saw Nomad so it was definitely worth re-evaluating (esp since it hit 1.0 recently). I wanted to try out Nomad to answer a personal curiosity: What does it do differently than Kubernetes? No better way than actually getting hands dirty, right?!

After following the brief tutorials from the official website I felt confident to try it for actual workloads. In my previous setup, I was hosting quite a few applications (Pihole, Gitea, Grafana etc) and thought it’ll be a nice way to learn how Nomad works by deploying the same services in the Nomad cluster. And I came in with zero expectations, I already had a nice setup which was reliable and running for me. My experience with a local Nomad cluster was joyful, I was able to quickly go from 0->1 in less than 30 minutes. This BTW is a strong sign of how easy Nomad is to get started with as compared to K8s. The sheer amount of different concepts you’ve to register in your mind before you can even deploy a single container in a K8s cluster is bizarre. Nomad takes the easy way out here and simplified the concepts for developers into just three things:

job
  \_ group
        \_ task

• Job: Job is a collection of different groups. Job is where the constraints for type of scheduler, update strategies and ACL is placed.
• Group: Group is a collection of different tasks. A group is always executed on the same Nomad client node. You’ll want to use Groups for use-cases like a logging sidecar, reverse proxies etc.
• Task: Atomic unit of work. A task in Nomad can be running a container/binary/Java VM etc, defining the mount points, env variables, ports to be exposed etc.

If you’re coming from K8s you can think of Task as a Pod and Group as a Replicaset. There’s no equivalent to Job in K8s. BUT! The coolest part? You don’t have to familiarise yourself with all different types of Replicasets (Deployments, Daemonsets, Statefulsets) and different ways of configuring them.

Want to make a normal job as a periodic job in Nomad? Simply add the following block to your existing Job:

periodic {
  cron = "@daily"
}

You want to make a service run as a batch job (on all Nomad nodes – the equivalent of Daemonset in K8s)? Simply make the following change to your existing job:

-type="service"
+type="batch"

You see this is what I mean by the focus on UX. There are many many such examples which will leave a nice smile on your face if you’re coming from K8s background.

I’d recommend reading Internal Architecture of Nomad if you want to understand this in-depth.

Architecture#

Tech stack for Hydra:

• Tailscale VPN: Serves as a mesh layer between my laptop/mobile and DO server. Useful for exposing internal services.
• Caddy for reverse proxying and automatic SSL setup for all services. I run 2 instances of Caddy:
  • Internal: Listens on Tailscale Network Interface. Reverse proxies all private services.
  • Public: Listens on DO’s Public IPv4 network interface. Reverse proxies all public-facing services.
• Terraform: Primary component to have IaC (Infra as Code). Modules to manage:
  • Cloudflare DNS Zone and Records
  • DO Droplet, Firewall rules, SSH Keys, Floating IPs etc.
  • Nomad Jobs. Used for running workloads after templating env variables, config files in Nomad job files.

Complexity of Nomad vs Kubernetes#

Nomad shines because it follows the UNIX philosophy of “Make each program do one thing well”. To put simply, Nomad is just a workload orchestrator. It only is concerned about things like Bin Packing, scheduling decisions.

If you’re running heterogeneous workloads, running a server (or a set of servers) quickly becomes expensive. Hence orchestrators tend to make sense in this context. They tend to save costs by making it efficient to run a vast variety of workloads. This is all an orchestrator has to do really.

Nomad doesn’t interfere in your DNS setup, Service Discovery, secrets management mechanisms and pretty much anything else. If you read some of the posts at Kubernetes Failure Stories, the most common reason for outages is Networking (DNS, ndots etc). A lot of marketing around K8s never talks about these things.

I always maintain “Day 0 is easy, Day N is the real test of your skills”. Anyone can deploy a workload to a K8s cluster, it’s always the Day N operations which involve debugging networking drops, mysterious container restarts, proper resource allocations and other such complex issues that require real skills and effort. It’s not as easy as kubectl apply -f and my primary gripe is with people who miss out on this in their “marketing” pitches (obvious!).

When to use Nomad#

Nomad hits the sweet spot of being operationally easy and functional. Nomad is a great choice if you want to:

• Run not just containers but other forms of workloads.
• Increase developer productivity by making it easier to deploy/onboard new services.
• Consistent experience of deployment by testing the deployments locally.
• (Not joking) You are tired of running Helm charts or writing large YAML manifests. The config syntax for Nomad jobs is human friendly and easy to grasp.

Nomad is available as a single binary. If you want to try it locally, all you need is sudo nomad agent -dev and you’ll have a Nomad Server, Client running in dev mode along with a UI. This makes it easy for the developers to test out the deployments locally because there’s very little configuration difference between this and production deployment. Not to forget it’s super easy to self-host Nomad clusters. I’m yet to meet anyone who self hosts K8s clusters in production without a dedicated team babysitting it always.

Once you eliminate the “blackbox” components from your stack, life becomes easier for everyone.

When to not use Nomad#

• If you’re relying on custom controllers and operators. Operator Pattern is a new way of managing large complex distributed systems (like databases, job queues etc). There are a lot of community built operators which help in reducing the effort to run these services. However, all of these are tied deeply into the “Kubernetes” ecosystem. If you find yourself running any of such operators, it’ll be tough (not impossible) to translate the same in Nomad ecosystem.

I genuinely cannot think of any other reason to not use Nomad!

Practical Scenarios#

Since I migrated a couple of workloads from my DO docker containers setup to Nomad, I’d demonstrate a few use cases which might be helpful if you want to start migrating your services to Nomad

Accessing a Web service with Reverse Proxy#

Context: I’m running Caddy as a reverse proxy for all the services. Since we discussed earlier, Nomad only is concerned about scheduling, so how exactly do you do Service Discovery? You need Consul (or something like Consul, Nomad has no hard restrictions) to register a service name with it’s IP Address. Here’s how you can do that:

In the .task section of your Nomad job spec, you need to register the service name with the port you’re registering and additional tags as metadata (optional):

service {
  name = "gitea-web"
  tags = ["gitea", "web"]
  port = "http"
}

Nomad’s template uses consul-template behind the scenes. This is a small utility which continuously watches for Consul/Vault keys and provides the ability to reload/restart your workloads if any of those keys change. It can also be used to discover the address of the service registered in Consul. So here’s an example of Caddyfile using Consul Template functions to pull the IP address of the upstream gitea-web service:

git.mrkaran.dev {
    {{ range service "gitea-web" }}
    reverse_proxy {{ .Address }}:{{ .Port }}
    {{ end }}
}

When a job is submitted to Nomad, a rendered template is mounted inside the container. You can define actions on what to do when the values change. For eg on a redeployment of Gitea container, the address will most likely change. We’d like Caddy to automatically restart with the new address configured in the Caddyfile in that case:

template {
  data = <<EOF
${caddyfile_public}
EOF

  destination = "configs/Caddyfile" # Rendered template.

  change_mode = "restart"
}

Using change_mode we can either send a signal or restart the task altogether.

Binding to different network interfaces#

I run a public instance of Gitea but I wanted to restrict the SSH access only to my Tailscale network. Nomad has an interesting feature host_network which lets you bind different ports of a task on different network interfaces.

network {
  port "http" {
    to = 3000
  }

  port "ssh" {
    to = 22

    # Need a static assignment for SSH ops.
    static = 4222

    # SSH port on the host only exposed to Tailscale IP.
    host_network = "tailscale"
  }
}

Templating Env Variables#

NOTE: This is not recommended for production.

Nomad doesn’t have any templating functionalities, so all the config must be sourced from Consul and secrets should be sourced from Vault. However in the time constraint I had, I wanted to understand Nomad and Consul better and use Vault at a later stage. I needed a way to interpolate the env variables. This is where Terraform comes into picture:

resource "nomad_job" "app" {
  jobspec = templatefile("${path.module}/conf/shynet.nomad", {
    shynet_django_secret_key   = var.shynet_django_secret_key,
    shynet_postgresql_password = var.shynet_postgresql_password
  })
  hcl2 {
    enabled = true
  }
}

We can pass the variables from Terraform (which can be sourced by TF_VAR_ in your local env) to the Nomad job spec. Inside the job spec we can use env to make it available to our task:

env {
  DB_PASSWORD              = "${shynet_postgresql_password}"
  DJANGO_SECRET_KEY        = "${shynet_django_secret_key}"
}

Running a backup job on the host#

I use restic to take periodic backups of my server and upload to Backblaze B2. Since Nomad supports running tasks as a different isolated environment (chroot) using exec driver and even without isolation using raw_exec driver, I wanted to give that a try. I’ve to resort using raw_exec driver here because /data file path on my host was not available to the chroot’ed environment.

job "restic" {
  datacenters = ["hydra"]
  type        = "batch"

  periodic {
    cron             = "0 3 * * *"
    time_zone        = "Asia/Kolkata"
    prohibit_overlap = true
  }
  ...
  task "backup" {
	  driver = "raw_exec"

	  config {
		# Since `/data` is owned by `root`, restic needs to be spawned as `root`. 

		# `raw_exec` spawns the process with which `nomad` client is running (`root` i.e.).
		command = "$${NOMAD_TASK_DIR}/restic_backup.sh"
	  }
  }
  ...
}

You can follow the rest of the config here.

Scope of Improvements#

Nomad has been an absolute joy to work with. However, I’ve spotted a few rough edge cases which I believe one should be aware of:

• host_network property sometimes gets ignored when doing a modification to service. I’ve opened an issue upstream but looks like other people are facing similar behaviours here and here.
• host_network as of present cannot bind to a floating IP address (DigitalOcean/GCP etc). I’ve to resort to using my droplet’s public IPv4 address for now.
• I tried using Consul Connect (service mesh with mTLS) but looks like again because of host_network, I’m unable to use it.
• Nomad CLI can definitely be improved for a much more consistent experience. I particularly missed using kubectl when using nomad.

That apart, I ended up sending a PR to upstream addressing a CLI arg ordering issue.

Gotchas:#

• On a Nomad server already bootstrapped, if you try changing server.bind_addr, it won’t have any effect. I almost pulled my hair debugging this, ultimately deleting the data_dir of the server resolved the issue for me.
• I’m running DB and the App together as a single “group” in my setup configs. Don’t do this in production. Whenever you restart the job, the group will restart both the containers. The side effect of this is pretty interesting: Since we use Consul to fetch the DB Host, the app may start before the DB boots up and registers its new address with Consul. I will fix the dependency in a future version but since I’m running fewer workloads and there are automatic retries, it’s okay enough for me to keep it like this.

Community#

Nomad’s community is pretty small compared to Kubernetes. However, the folks are super responsive on Gitter, Discourse and Github Issues. A few noteworthy mentions:

• @the-maldridge helped me with my doubts in Gitter.
• @tgross who is super responsive on Github issues and does an excellent job at housekeeping the issues.
• @shantanugadgil who is also pretty active in the community.

Nomad’s ecosystem is still in its nascent stage and I believe there are a lot of contribution opportunities for folks interested in Golang, Ops, Distributed Systems to contribute to Nomad. The codebase of Nomad is approachable and there are quite a few key areas which can be contributed to:

• Docs: More examples, practical use cases.
• Nomad Job files: There are many helm charts available to follow best practices. Something similar in Nomad will definitely be interesting.
• Nomad Gotchas: Since K8s is widely used and has a much larger adoption, it’s only natural that the failure stories of K8s are highlighted a lot. Nomad being a pretty smaller community, we need more debugging and “things that went wrong” reference materials. You learn more from failures than 101 setup guides :)

Final Thoughts#

I think I’m sold on Nomad. I’ve used Kubernetes in prod for 2 years but if you were to ask me to write a Deployment spec from scratch (without Googling/kubectl help) I won’t be able to. After writing Nomad configs, I just can’t think of the sheer amount of boilerplate that K8s requires to get an application running.

Nomad is also a simpler piece to keep in your tech stack. Sometimes it’s best to keep things simple when you don’t really achieve any benefits from the complexity.

Nomad offers less than Kubernetes and it’s a feature, not a bug.

Fin!

Discussions#

• HackerNews
• Lobster
• Twitter

]]> Karan Sharma https://nadh.in/blog/the-atmanirbhartha-of-open-source-software/ https://nadh.in/blog/the-atmanirbhartha-of-open-source-software/ Sat, 30 Jan 2021 00:00:00 GMT In the Indian startup circles, Atmanirbhar (self-reliance) is the word of the year. Technology startups of all shapes and sizes, “unicorns” and non-unicorns have incorporated the tri colour and the Made in India label into their brand messaging and advertising campaigns—marketing prowess and valuations built on top of rapid innovation enabled by Free and Open Source Software (FOSS) created all over the world by countless programmers volunteering their time and effort writing code for everyone to solve problems and build enterprises. Linux, the quintessential example of FOSS, if quantified, would turn out to have created trillions of dollars in value for humanity.

]]> Kailash Nadh https://mrkaran.dev/posts/gitlab-runner-ecr/ https://mrkaran.dev/posts/gitlab-runner-ecr/ Fri, 29 Jan 2021 02:40:55 GMT There are some things you expect to just work. Sadly trying to make Gitlab Runner with AWS ECR turned out to be quite a daunting task and the little documentation in this area doesn’t help. There’s even a 4 years old issue and everyone there is echoing the sentiment that this is unnecessarily a lot harder than it should have been.

Anyway, since I spent a lot of time figuring out how to make a Private Registry work with a cross-account ECR, I’m documenting these steps hoping it’ll help someone someday :).

The Problem#

There are mainly 2 seemingly same but different problems when it comes to using ECR. Let’s discuss both of them separately:

• Pulling a private image from ECR using the Docker Executor. For eg, if your gitlab-ci.yml looks like:

test-pull:
  image: $PRIVATE_ECR_IMAGE
  script:
    - echo "Hello World!"

In this case, the Docker Executor needs to be “authenticated” to AWS ECR so that it can pull $PRIVATE_ECR_IMAGE.

• Pulling a private image inside the job. For eg, if you’re using Kaniko:

docker-build:
  image: gcr.io/kaniko-project/executor:debug
  script:
    - |
      /kaniko ...
      # Inside this step, we use a PRIVATE_ECR_IMAGE defined in our `Dockerfile`.

In this case, Kaniko needs to be “authenticated” to AWS ECR so that it can pull $PRIVATE_ECR_IMAGE.

NOTE I prefer Kaniko over DIND as it is faster, doesn’t require running the privileged container, caching is simplified, and is in general a lot simpler to setup.

The Solution#

So, for the first case, where you want to authenticate the Docker Executor to AWS ECR, you’ll need 2 things:

1. Setup DOCKER_AUTH_CONFIG environment variable to { "credsStore": "ecr-login" } in the config.toml of the runner. For eg:

[[runners]]
  name = "Test"
  url = "https://gitlab.internal/"
  token = "REDACTED"
  executor = "docker"
  environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"]

2. Now, we’ve specified the Credential Store for Docker, but we don’t have this binary docker-credential-ecr-login in our runner. AWS provides amazon-ecr-credential-helper which is a neat way of automatically authenticating with AWS ECR based on your Access Keys/IAM role. What does automatic mean here? So, the normal docker login is a basic auth command, where if you’ve to log in to ECR, you need to do something like:

aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com

This is problematic because the authorization token is valid for 12 hours. Further, you’ve to log in to multiple registry IDs separately. Managing this is a nightmare, so Docker instead of just relying on Basic Auth, came up with a neat mechanism: docker-credential-helpers. This allows you to keep your secret tokens in your Keystore. A new credential helper can be written in Go which implements the credentials.Helper interface. This is what amazon-ecr-credential-helper does by offering various ways like AWS IAM Roles, Assumed Roles, Access Keys, etc to authenticate with ECR.

This is where I stumbled the most. I downloaded the binary from the Github Releases but this binary is statically compiled with muslc libraries.

However gitlab/gitlab-runner is based on the ubuntu docker image, so the above binary never worked. The strangest thing was the unhelpful error message that sh returned as explained in this post.

To make things easier, I baked my own gitlab-runner image with the above binary compiled inside the image using go get:

FROM ubuntu:20.04 AS build
ENV DEBIAN_FRONTEND=noninteractive 
RUN : \
 && apt-get update \
 && apt-get install --no-install-recommends -y git golang-go ca-certificates \
 && rm -rf /var/lib/apt/lists/* \
;
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
WORKDIR /build
RUN mv /root/go/bin/docker-credential-ecr-login .

FROM gitlab/gitlab-runner:v13.8.0 AS deploy
COPY --from=build /build/docker-credential-ecr-login /usr/local/bin/docker-credential-ecr-login

The above image bakes in the docker-credential-ecr-login binary and also puts it under /usr/local/bin so it’ll be available under $PATH to the Docker engine.

With the above 2 things, if the runner’s server (EC2 instance/K8s pod) has access to the ECR image, it should be able to pull.

Now coming to the 2nd problem, where we wanted kaniko to authenticate to ECR, things are a bit simpler:

• Kaniko comes with docker-credential-ecr-login baked in. All you need to do is add the following to ~/.docker/config.json as explained here.

{ "credsStore": "ecr-login" }

Next, we need to mount the AWS Credentials to Kaniko’s image so that it can use AWS SDK to perform a login to ECR. We do that by using volumes of the runner:

[runners.docker]
volumes = ["/home/ubuntu/runners/test/aws-credentials:/root/.aws/credentials:ro"]

This mounts the aws-credentials file from the host inside the container which the runner spawned (kaniko in this case).

A sample aws-credentials file if you’re using a cross-account access can look like:

[default]
role_arn=arn:aws:iam::ACCOUNT_ID:role/assume-role-{{ROLE_NAME}}
credential_source=Ec2InstanceMetadata
region=ap-south-1

You can put in your normal AWS Keys or leave them blank if you want to use your IAM Role. With mounting the AWS Credentials inside Kaniko’s container, you can authenticate to cross-account ECRs as well (once you set up the whole assumed-role/trusted entities flow).

Sample Runner#

If you want to take a look at how the complete flow looks like:

• Start the Runner service using docker-compose:

version: '3.7'

services:

  test:
    # As explained above, this image has `docker-credentials-ecr-login` baked in.
    image: {{ACCOUNT_ID}.dkr.ecr.ap-south-1.amazonaws.com/custom/gitlab-runner:13.8.0
    restart: always
    volumes:
      # Automatically created; config.toml.
      - './test/runner-config:/etc/gitlab-runner'
      # Mount AWS Credentials for Docker Executor to authenticate.
      - './test/aws-credentials:/root/.aws/credentials:ro'
      # Mount Docker Socket so that executor can communicate with it.
      - '/var/run/docker.sock:/var/run/docker.sock'

• Register a new runner:

docker-compose exec test register

Fill in the basic info and edit ./test/runner-config.toml with the following options:

[[runners]]
  name = "Test"
  executor = "docker"
  environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"]
  [runners.docker]
    volumes = ["/home/ubuntu/runners/test/aws-credentials:/root/.aws/credentials:ro"]

Conclusion#

Honestly, this was a lot of trial and error to figure out how to use private images with Gitlab. Some important links and references that helped me figure this out:

• https://gitlab.com/bmares/gitlab-runner-ecr-auth-example/
• https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1583#note_84649153

Fin!

]]> Karan Sharma https://saket-choudhary.me/blog/2020/12/27/scvi/ https://saket-choudhary.me/blog/2020/12/27/scvi/ Sat, 26 Dec 2020 18:30:00 GMT scvi-tools exists as a suite of tools for performing dimensionality reduction, data harmonization, and differential expression. One key advantage of using scvi-tools is that it inherently supports loading and training data in mini-batches and hence is practically infinitely scalable (Lopez et al., 2018).

scvi-tools uses generative modeling to model counts originating from a scRNA-seq experiment with different underlying models catering to other experiments. “Generative modeling” is a broad term that implies models of distributions $P(X)$ , defined over some collection of datapoints $X$ that exist in a high dimensional space. In scRNA-seq, each datapoint corresponds to a cell $c$ which has a multidimensional vector $X_{c,g} \in \mathcal{R}^{20000}$ containing read counts or UMIs of 20000 genes. A scRNA-seq datasets contains not one but a few thousand if not millions of cells. The generative model’s task is to capture the underlying representation of these cells. “Representation” here is a loose term, but more formally given a $\text{gene} \times \text{cell}$ matrix whose distribution $P_{\text{truth}}(X)$ is unknown, the generative model tries to learn a distribution $P(X)$ which is as close to $P_{\text{truth}}(X)$ as possible.

In order to obtain $P(X)$ , the model should be able to exploit the underlying structure in data. Neural networks are powerful functional approximators given their ability to capture non-linearities. Variational autoencoders utilize neural networks to build generative models that can approximate $P_{truth}(X)$ in a decently quick fashion. The reason this works is because any $d$ dimensional distribution can be approximated by starting with $d$ gaussian random variables and passing them through a complicated function (Devroye, 1986). A famous example of this is generating a 2D circle from a 2D gaussian blob.

Figure 1. A 2D gaussian blob can be passed through a sufficiently complicated function $g(z) = \frac{z}{\alpha} + \frac{z}{||z||}$ to obtain a 2D ring.

scvi-tools also starts from a gaussian random variable and propogates it through its various layers such that the output count for a gene and a particular cells is close to its observed value. It does it over four main steps:

1. Generate a gaussian

2. Pass the gaussian through a neural network to approximate gene-cell proportions ($\rho_{g,c}$)

3. Generate a count $y_{c,g}$ for each gene-cell using the estimated proportion in step 2 and and the total sequencing depth along with an estimated dispersion $\phi_g$.

4. Calculate reconstruction error between generated count $y_{c,g}$ and observed count $x_{c,g}$

The aim is to minimize the reconstruction error in step 4 by optimizing the neural network weights and the estimated parameters $\rho_{c,g}$ and $\theta_g$.

$\begin{aligned} {\color{purple}z_c} &\sim \mathcal{N}(0,I) & \text{\color{purple}Cell embedding} \\ {\color{red}\rho_{g,c}} &\sim \text{softmax}(f_w(z_c)) & \text{\color{red}Normalized expression } \\ y_{c,g} &\sim \text{NB}({\color{blue} l_c} {\color{red}\rho_{c,g}}, \phi_g) & \text{Observed counts} \end{aligned}$

The total sequencing depth for a cell can be also be learned by the network inherently, but the latest version (0.8.0) of scVI supports using observed library size. I started using observed library sizes before it became part of the implementation. The training time is faster and in my limited testing, the downstream clustering results look slightly better with using observed library size, but it could also be due to other reasons.

The latent distribution $Z$ thus learned is a reduced dimensional latent reprsentation of the data. I will use [PBMC3k dataset] (https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3k) for all the analysis here. We can do a UMAP visualization and the clusters tend to match up pretty well with ground truth, though there is possiblity of improvement.

Figure 1. UMAP on the latent representation learned by scvi on PBMC3k dataset.

We now have a $P(Y)$ and access to all intermediate values we can do a ton of things. But the first thing would be to check if $P(Y)$ is indeed correct. One such way of performing validity checks on this model is posterior predictive checks (PPC). I learned of PPCs through Richard McElreath’s Statistical Rethinking (McElreath, 2020), which forms an integral part of all his discussions.

The idea of a PPC is very simple: simulate replicate data from the learned model and compare it to the observed. In a way you are using your data twice, to learn the model and then using the learned model to check it against the same data. A better designed check would be done on held out dataset, but it is perfectly valid to test the model against the observations used to train the model.

The simplest checks for scRNA-seq counts is the mean-variance relationships. The simulated means and variances from the learned model should match that of the observed data on both a cell and a gene level.

Figure 2. Comparison of generated against observed means, variance and mean-variance relationships.

The simulated mean-variance relationship aligns very well with the observed relationship.

Let’s compare how the dispersion looks like:

Figure 3. Comparison of generated against observed means, variance and mean-variance.

Variation with gene detection rate:

Figure 4. Comparison of generated against observed means, variance and mean-variance.

The loss function being minizmied to infere the parameters minimizes the reconstruction loss between generated counts $X$ and observed counts $Y$.

Figure 5. Comparison of generated against observed means, variance and mean-variance.

One thing I still need to wrap around my head is how much informative the reconstruction error itself is. For example, a UMAP of this reconstruction error mimics that of the latent representation:

Figure 5. Comparison of generated against observed means, variance and mean-variance.

1. Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018). Deep generative modeling for single-cell transcriptomics. Nature Methods, 15(12), 1053–1058.
2. Devroye, L. (1986). Sample-based non-uniform random variate generation. Proceedings of the 18th Conference on Winter Simulation, 260–265.
3. McElreath, R. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan. CRC press.
4. Ma, S., Zhang, B., LaFave, L. M., Earl, A. S., Chiang, Z., Hu, Y., Ding, J., Brack, A., Kartha, V. K., Tay, T., & others. (2020). Chromatin potential identified by shared single-cell profiling of RNA and chromatin. Cell, 183(4), 1103–1116.

]]> Saket Choudhary https://mrkaran.dev/posts/job-queue-golang/ https://mrkaran.dev/posts/job-queue-golang/ Tue, 01 Dec 2020 02:40:55 GMT In this post we’ll see how to create a simple job queue in Golang. There are tonnes of libraries and posts out there doing overly complicated stuff, however if your need is pretty minimal or you want to understand the concepts from ground up, this post aims to do just that and nothing more.

We’ll be using concepts like WaitGroup, Channels and Contexts to build our own Job Queuing mechanism. It primarily involves 2 components:

• Queue: A queue which has a list of items waiting to be processed.

• Worker: A worker constantly listening to that queue and processing the events as desired.

With these 2 main ideas behind us, let us create our sample structure:

package dispatch

type Dispatcher interface {
	// Push takes an Event and pushes to a queue.
	Push(Event) error
	// Run spawns the workers and waits indefinitely for
	// the events to be processed.
	Run()
}

// EventDispatcher represents the datastructure for an
// EventDispatcher instance. This struct satisfies the
// Dispatcher interface.
type EventDispatcher struct {
	Opts     Options
	Queue    chan models.Notification
	Finished bool
}

// Options represent options for EventDispatcher.
type Options struct {
	MaxWorkers int // Number of workers to spawn.
	MaxQueueSize int // Maximum length for the queue to hold events.
}

// NewEventDispatcher initialises a new event dispatcher.
func NewEventDispatcher(opts Options) (Dispatcher) {
	return EventDispatcher{
		Opts: opts,
		Queue: make(chan Event, opts.MaxQueueSize),
		Finished: false,
	}
}

Pushing to Queue#

Now that we have our basic structure ready, let’s write a function to push events to queue.

A queue is simply a channel. We have created a new queue of size MaxQueueSize while initialising the EventDispatcher.

Queue: make(chan Event, opts.MaxQueueSize)

To push events into it, we’ll simply do: d.Queue <- event. This adds a new item (event) of type Event to our queue.

// Push adds a new event payload to the queue.
func (d *EventDispatcher Push(event Event) error {
	if d.Finished {
		return errors.New(`queue is closed`)
	}
	d.Queue <- event
	return nil
}

Listening to Queue#

So the client is calling Push() on our EventDispatcher and events are being pushed in the channel. But there’s no one reading from this channel so far. Let’s fix that by spawning workers, who will listen on the channel indefinitely and process the events:

for {
    select {
        case event <- d.Queue:
        event.Process()
    }
}

In the above snippet, we are simply looping indefinitely to scan through all items in the queue. event <- d.Queue is basically fetching the item from the channel and assigning a value to it.

event.Process() is a dummy function but it basically indicates that whatever processing that needs to be done should be handled here.

Right now, you’ll be wondering two things:

• If this is an infinite loop, how do we guarantee it runs forever?
• How do I spawn more workers if I need concurrency?

To address these problems, let’s add in WaitGroups and GoRoutines to our mix.

WaitGroups will help us keep a count of workers which have been spawned and until each one of them finishes processing, wait groups will keep blocking indefinitely using wg.Wait().

And to bring in more workers, we’ll simply spawn them with GoRoutines:

go func() {
	for {
	  select {
	    case event <- d.Queue:
		event.Process()
	  }
	}
}()

Now, spawning n Goroutines is just a matter of a simple for loop over this:

for i:=0; i<d.Opts.MaxWorkers; i++{
	wg.Add(1) // Add a wait group for each worker
	go func() {
		for {
		select {
			case event <- d.Queue:
			event.Process()
		}
		}
	}()
}

Perfect! But hang on! We have missed a critical thing. How do we handle cancellations? For eg, when your program shuts down, we should clean up all the Goroutines spawned and process the remaining messages in queue. For that, we need to listen to a Cancellation channel. The only purpose of this channel is to listen for SIGINT or SIGTERM signals and whenever either of them is received, we should flush our events.

Here’s how the client would initialise a context:

// Create a channel to relay `SIGINT` and `SIGTERM` signals.
closeChan := make(chan os.Signal, 1)
signal.Notify(closeChan, os.Interrupt, syscall.SIGTERM)
ctx, cancel := context.WithCancel(context.Background())

And in the main thread, the client would block on closeChan channel like:

// Listen on close channel indefinitely until a
// `SIGINT` or `SIGTERM` is received.
<-closeChan
// Cancel the context to gracefully shutdown.
cancel()

When cancel() is called, it does something special. It passes a value to ctx.Done() channel. We can listen to this channel in the .Run() function and flush pending events accordingly:

case <- ctx.Done():
	// Ensure no new messages are added.
	d.Finished = true
	// Flush all events.
	e.Flush()
	// This Goroutine has finished processing.
	wg.Done()

---

Stitching all pieces together, we finally have:

package dispatch

type Dispatcher interface {
	// Push takes an Event and pushes to a queue.
	Push(Event) error
	// Run spawns the workers and waits indefinitely for
	// the events to be processed.
	Run()
}

// EventDispatcher represents the datastructure for an
// EventDispatcher instance. This struct satisfies the
// Dispatcher interface.
type EventDispatcher struct {
	Opts     Options
	Queue    chan models.Notification
	Finished bool
}

// Options represent options for EventDispatcher.
type Options struct {
	MaxWorkers int // Number of workers to spawn.
	MaxQueueSize int // Maximum length for the queue to hold events.
}

// NewEventDispatcher initialises a new event dispatcher.
func NewEventDispatcher(opts Options) (Dispatcher) {
	return EventDispatcher{
		Opts: opts,
		Queue: make(chan Event, opts.MaxQueueSize),
		Finished: false,
	}
}

// Push adds a new event payload to the queue.
func (d *EventDispatcher Push(event Event) error {
	if d.Finished {
		return errors.New(`queue is closed`)
	}
	d.Queue <- event
	return nil
}

// Run spawns workers and listens to the queue
// It's a blocking function and waits for a cancellation
// invocation from the Client.
func (d *EventDispatcher Run(ctx context.Context) {
	wg := sync.WaitGroup{}
	for i := 0; i < d.Opts.MaxWorkers; i++ {
		wg.Add(1) // Add a wait group for each worker
		// Spawn a worker
		go func() {
			for {
				select {
				case <-ctx.Done():
					// Ensure no new messages are added.
					d.Finished = true
					// Flush all events
					e.Flush()
					wg.Done()
					return
				case e <- d.Queue:
					e.Process()
				}
			}
		}()
	}
	wg.Wait()
}

// Push adds a new event payload to the queue.
func (d *EventDispatcher Push(event Event) error {
	if d.Finished {
		return errors.New(`queue is closed`)
	}
	d.Queue <- event
	return nil
}

This post doesn’t cover how to flush or process the events as these are implementation specific details. This is a pretty barebones structure and you can modify the code according to your usecase.

Fin!

]]> Karan Sharma https://captnemo.in/blog/2020/11/29/fantastic-beasts-graffiti/ https://captnemo.in/blog/2020/11/29/fantastic-beasts-graffiti/ Sun, 29 Nov 2020 00:00:00 GMT “Fantastic Beasts and Where to Find Them” is a curious book:

• The in-universe book is written by Newt Scamander and was published in 1927.
• The first edition of the companion book was published in 2001. This is apparently the 52nd in-universe edition with a foreword from Dumbledore and was released to the muggle world for charity.
• The 2001 edition pretends to be Harry’s copy of the book as of the end of Harry’s 4th year at Hogwarts. As such it includes hand-written comments from the trio. (Yes, Hermione writes on books!)

However, with the release of the film of the same name in 2016 - a new edition was released with lots of changes:

1. 6 new beasts that made an appearance in the film were added to the book¹:
   • Hidebehind
   • Hodag
   • Horned Serpent
   • Snallygaster
   • Thunderbird
   • Wampus cat
2. The hand-lettering was removed.
3. Dumbledore’s foreword is removed from the book, in favor of a in-universe foreword from Newt Scamander.
4. “About the Author” section changes Newt’s background. He no longer graduates from Hogwarts, just “leaves” it, as portrayed in the film.

All of these changes are meant to fix the inconsistencies in the book with the canon, however that also makes the book much less charming. I got myself a copy of the Hogwarts Library boxset a few years ago, which includes the newer edition of the book (Bloomsbury) - that means no witty comments from Ron.

Since it didn’t have the hand-lettering, I took it upon myself to fix that mistake. Thankfully, lists of all the comments in 2001 edition are available on the internet. The trickiest part was the “this book belongs to” page, which is missing from the newer edition. I ended up creating a faux-library card for that instead.

Here is what it looks like:

• 
  Ron plays hangman and loses.
• 
  Hermione writes on books!
• 
• 
• 
• 
• 
• 

Thanks to Bhavya for helping with the troll illustration.

1. I didn’t like the new additions, they sound less like a textbook and more like a transcript of what happened in the film. ↩

]]> Nemo https://mrkaran.dev/posts/analysis-paralysis/ https://mrkaran.dev/posts/analysis-paralysis/ Fri, 27 Nov 2020 02:40:55 GMT Have you ever been stuck in a situation where you’ve spent countless hours digging through lot of options, overthinking their pros/cons and having a tough time to decide?

If yes, I’ve been in the same boat myself and the more I think about this, I think it’s causing some serious issues in my productivity. Since I’ve started to self host a lot of services, I’m overthinking and overanalysing the software choices available to do a certain task. For eg, a personal wiki, sounds such a simple and common use case but I’ve personally tried countless choices. I’ve read and searched for these discussions on lobster/HN/Reddit discussions boards and that has been a real time sink for me, now that I look back. I’ll try an option for maybe a week or two at best and then jump on to something else in search of the perfect solution.

It’s hard for me to stick to a solution. And it’s even more embarassing when you tell your friends about the “latest new distro” you’re using, announce it to the world on Twitter and when someone asks, “hey how’s that OS working for you” after a month, you tell them “Oh, I switched from it last week”. I’m not making any of this up, I’ve been in these shoes myself and honestly it’s not a good feeling.

In the search of perfect I tend to lose sight of things that are important. Things which are slow, boring but they work are more reliable than a software which changes fast and breaks often. I want to fail cheap and quickly rather than spending a lot of time and still eventually failing.

No, this post doesn’t have any answers on how to fix this problem. Well, if I myself knew then I’d not be writing this but giving a TEDx talk on it. These are just my observations and something that I intend to actively fix.

Boring is better. I want to stick to stable choices when it comes to trivial things in life. I don’t wanna seek excitement or joy from switching to a cool new distro. Instead, by actually doing some important project at my work or building my side projects, something that gives me a genuine feeling of happiness. After all grass always seems to look greener on the other side.

I don’t usually pen down my thoughts like these, but I want to refer to this public post as a reminder for myself the next time I’m about to make the same mistake.

Fin.

UPDATE (2021-11-02): There’s a really nice video I came across today on the same subject. I’d highly recommend you to give this a listen if you found the above post relatable.

]]> Karan Sharma https://mrkaran.dev/posts/migrating-to-zola/ https://mrkaran.dev/posts/migrating-to-zola/ Sun, 08 Nov 2020 02:40:55 GMT CSS Processors. What took me time however was to figure out how to get opengraph tags in each page. Hugo provides nifty template for this use case but Zola is pretty barebones like that. People who care a lot about SEO need to spend some extra efforts here. Future# Zola is still a pretty new kid on the block but the author shares the same frustration about Hugo: it personally drives me insane, to the point of writing my own template engine and static site generator. Yes, this is a bit biased. – Source This also reflects in the issues/PRs I’ve seen for Zola and the author is opinionated about not adding features which would make Zola complicated. Overall I am very happy with the switch and it was long due. I feel more confident in tweaking certain sections of my website. I plan to open-source the current theme in the next few days. You can read the Source Code of this website if you’d like to explore how this website is built. Fin!]]> I’ve been writing on this blog for about 2 years now. This has been the longest I’ve stuck on to the same technology stack for my blog. I’ve previously jumped from a Jekyll based static site to a Medium blog before finally settling for Hugo.

I’ve been using Hugo since 2018 but I don’t recall as to why I went ahead with it. Maybe it was increasingly popular at that time and everyone touted Hugo as the solution to Static Site Generator (referred to as SSG from here on). There are 1000s of SSGs and at least a dozen of websites which lists all the SSGs out there. This is crazy by any standards. Hugo started as a generic blog generator but over the years it has become a website generator. It’s no longer aimed at people who just want to have a small little static website/blog but supports all the use cases for people building full-fledged static websites. IMHO these two goals are overarching however this has resulted in a simple project to become incredibly complex over time.

Tipping Point#

Anyway, so I wanted to change the look of the homepage on my website so I decided to look at Hugo’s documentation. Hugo’s documentation is great for someone who knows what exactly are they looking for. The documentation is so huge that you simply cannot grok it in one evening. I had zero ideas on how to customise the damn homepage of my blog and after spending hours buried in the documentation I was able to kind of figure the solution but it was unintuitive, to say the least. Apparently, to override any template from the theme, you have to mirror the directory structure of the theme in your root directory. Which meant, I needed to look at the source code of the theme, figure out the project structure, copy-paste all the folder names and put my override of index.html there. Which, BTW magically overrides it. This whole magic thing is BS and I am being strongly opinionated here.

There is more than one way to do something in Hugo. Different theme authors use different styles, which makes the whole thing even more complex. It also means for my customisations to work across themes, well you guessed it right: it’s impossible.

Recently I discovered that I was unable to preview my Hugo website locally without internet because I had a Twitter shortcode in one of my blog post (which makes an API call to Twitter to render a nice card preview). The site completely failed to render instead of just logging a warning. Bollocks.

The tipping point for me, however, was when the theme I was using stopped working with the latest version of Hugo at that point. So, picture this – You make dozens of custom changes and then one update just breaks your website. Now not only you have to fix your shit but the theme you were using, you’ve to make upstream changes to the theme or maintain your own fork. And no, this is not a one-off experience. Hugo upgrades are a joke, they are known to break very very often.

I was done at this point. I didn’t want to deal with this BS of continuously fighting the generator for my blog.

A fresh change#

Being a practitioner of Yak Shaving, I discussed the idea of a “tinyhugo” with Kailash and Sarat. We’d arrived at a spec and I started writing some code to pander to my NIH syndrome.

However, I was still not convinced that a simpler solution doesn’t exist. I spent countless hours exploring other alternatives. I’d used Lektor, Pelican, Eleventy before finally stumbling upon Zola from HN/Lobster discussions. I’ve got to say, the landing page gave a fresh feeling - one that I’ve not seen with any other alternatives. In fact quite opposite to the Eleventy landing page which looks like an over-engineered piece of software to generate websites (Not hating on it, there might be use cases for it, but the JS tooling and dependency system is something that I would not want to touch with a 10ft pole).

Zola’s primary appeal to me was that like Hugo it’s extremely fast and comes as a single binary no dependency package. I looked at the docs the first impression was they are concise enough to get a basic idea. Zola is strongly opinionated, even to the extent of dictating a project structure and sometimes filenames too. I actually preferred this over the magic Hugo does. In less than 2 hours I was able to port the home page of my blog (and tweak it to my liking) in Zola. I decided to abandon my own tinyhugo attempt because for the very fact Zola fits my needs very well.

The thing that I really loved about Zola is how it enforces a separation between Section and Pages. The section represents a “collection” of posts. So a blog can be a section, and I can have another section called “Book Reviews”. I could easily tell Zola where to look for the templates by specifying the same in content/book_reviews/_index.md. I don’t have to read Hugo docs or do Google-fu to figure this out, it’s right there in the docs and very apparent.

For the record, I still don’t know how to customise different templates for different sections in Hugo, but I couldn’t care less.

Migration#

The migration was pretty straightforward – I had to copy the content folders of my blog (which are just a bunch of .md files) and replace YAML frontmatter to TOML. There were a few variable changes that I needed to do manually but since they were a manageable 20-25 posts, I did it by hand. I could potentially automate but then rabbit deep in the rabbit hole of Yak Shaving. The good part was that I was able to retain the same URL structure for my new blog because the URL scheme was based on the file paths.

I spent some time porting hugo-ink to Zola and did minor CSS tweaks to it. Zola uses the Tera language for templating and it’s much more pleasing to eyes than the Go Template syntax. Zola comes with pretty neat features like Search, RSS/Atom Feeds, Syntax Highlighting and SASS->CSS Processors.

What took me time however was to figure out how to get opengraph tags in each page. Hugo provides nifty template for this use case but Zola is pretty barebones like that. People who care a lot about SEO need to spend some extra efforts here.

Future#

Zola is still a pretty new kid on the block but the author shares the same frustration about Hugo:

it personally drives me insane, to the point of writing my own template engine and static site generator. Yes, this is a bit biased. – Source

This also reflects in the issues/PRs I’ve seen for Zola and the author is opinionated about not adding features which would make Zola complicated. Overall I am very happy with the switch and it was long due. I feel more confident in tweaking certain sections of my website. I plan to open-source the current theme in the next few days.

You can read the Source Code of this website if you’d like to explore how this website is built.

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/terraform-route53-import/ https://mrkaran.dev/posts/terraform-route53-import/ Sun, 18 Oct 2020 02:40:55 GMT data/company-tld.json # Loads the zone records in a dict def load_records(zone_file=ZONE_FILE): with open(zone_file) as record_file: data = json.load(record_file) return data 2. Import the record in Terraform state.# To do this, Terraform CLI comes with an import command. However for import to work, you need to have a resource declaration in your Terraform file already. From the official documentation: Because of this, prior to running terraform import it is necessary to write manually a resource configuration block for the resource, to which the imported object will be mapped. To overcome this restriction, we will create a dummy.tf and programatically write the configuration block for each record. # Writes the dummy Terraform template which is required # before `terraform import` runs. def template_dummy_file(resource_name): add_dummy_record = Template( """ resource "aws_route53_record" "$resource_name" { # (resource arguments) } """ ) dummy_file_path = path.join(TERRAFORM_DIR, "dummy.tf") with open(dummy_file_path, "a") as f: f.write(add_dummy_record.substitute(resource_name=resource_name)) AWS Route53 module can import aws_route53_record as decsribed here. We will run this command as a subprocess. # Shells out `terraform import` command in the host OS. def terraform_import(resource_name, resource_type): import_command = f"terraform import aws_route53_record.{resource_name} {ZONE_ID}_{resource_name}_{resource_type}" run(import_command, shell=True, check=True) 3. (Optional) Move Resources in a Module# In case you are using a Module to manage AWS Route53 resources, you’ll need to move the declaration from resource to module configuration block. This is described more in detail here. The module declaration/naming would depend on how the module is configured. To demonstrate, the module I use internally requires the name to be of the format resource_name-resource_type. To achieve this, you can call terraform state mv as a subprocess: # Shells out `terraform state mv` command in the host OS. def terraform_move(resource_name, resource_type): mv_command = f"terraform state mv aws_route53_record.{resource_name} 'module.{MODULE_NAME}.aws_route53_record.route53_record[\"{resource_name}-{resource_type}\"]'" run(mv_command, shell=True, check=True) That’s it! Running terraform plan should now show you the changes and if you imported every record correctly you should not see any drift from the real world state. You can view the entire script here: import json from os import getenv, path from string import Template from subprocess import run from sys import exit ZONE_ID = getenv("ZONE_ID") MODULE_NAME = getenv("MODULE_NAME") ZONE_FILE = getenv("ZONE_FILE") TERRAFORM_DIR = getenv("TERRAFORM_DIR") # Returns the variable key if not present in ENV. def check_env_vars(): if not ZONE_FILE: return "$ZONE_FILE" if not ZONE_ID: return "$ZONE_ID" if not MODULE_NAME: return "$MODULE_NAME" if not TERRAFORM_DIR: return "$TERRAFORM_DIR" return "" # Loads the zone records in a dict def load_records(zone_file=ZONE_FILE): with open(zone_file) as record_file: data = json.load(record_file) return data # Writes the dummy Terraform template which is required # before `terraform import` runs. def template_dummy_file(resource_name): add_dummy_record = Template( """ resource "aws_route53_record" "$resource_name" { # (resource arguments) } """ ) dummy_file_path = path.join(TERRAFORM_DIR, "dummy.tf") with open(dummy_file_path, "a") as f: f.write(add_dummy_record.substitute(resource_name=resource_name)) # Shells out `terraform import` command in the host OS. def terraform_import(resource_name, resource_type): import_command = f"terraform import aws_route53_record.{resource_name} {ZONE_ID}_{resource_name}_{resource_type}" run(import_command, shell=True, check=True) # Shells out `terraform state mv` command in the host OS. def terraform_move(resource_name, resource_type): mv_command = f"terraform state mv aws_route53_record.{resource_name} 'module.{MODULE_NAME}.aws_route53_record.route53_record[\"{resource_name}-{resource_type}\"]'" run(mv_command, shell=True, check=True) if __name__ == "__main__": missing = check_env_vars() if missing: exit(f"Required env variable {missing} is missing.") records = load_records() for i in records.get("ResourceRecordSets"): resource_name = i.get("Name") resource_type = i.get("Type") template_dummy_file(resource_name) terraform_import(resource_name, resource_type) terraform_move(resource_name, resource_type) print(f"Imported {resource_name}") Hope this tiny Python script helps you transition your AWS Route53 records neatly and effortlessly! Fin!]]> Terraform has a straightforward way of importing existing records (managed outside Terraform) via terraform import command. The usage is documented here and works well if you have a handful of records to import. However when you work with custom Terraform modules and have a whole bunch of records to be imported, you’d look out ways to script the entire workflow. I did this a few weeks back at work and thought to share a solution which works well for my usecase.

How it works#

The task consists of 3 parts:

1. Import all existing records in a hosted zone using AWS CLI.#

aws route53 list-resource-record-sets --hosted-zone-id XXX > data/company-tld.json

# Loads the zone records in a dict
def load_records(zone_file=ZONE_FILE):
    with open(zone_file) as record_file:
        data = json.load(record_file)
    return data

2. Import the record in Terraform state.#

To do this, Terraform CLI comes with an import command. However for import to work, you need to have a resource declaration in your Terraform file already.

From the official documentation:

Because of this, prior to running terraform import it is necessary to write manually a resource configuration block for the resource, to which the imported object will be mapped.

To overcome this restriction, we will create a dummy.tf and programatically write the configuration block for each record.

# Writes the dummy Terraform template which is required
# before `terraform import` runs.
def template_dummy_file(resource_name):
    add_dummy_record = Template(
        """
	resource "aws_route53_record" "$resource_name" {
	# (resource arguments)
	}
	"""
    )
    dummy_file_path = path.join(TERRAFORM_DIR, "dummy.tf")
    with open(dummy_file_path, "a") as f:
        f.write(add_dummy_record.substitute(resource_name=resource_name))

AWS Route53 module can import aws_route53_record as decsribed here. We will run this command as a subprocess.

# Shells out `terraform import` command in the host OS.
def terraform_import(resource_name, resource_type):
    import_command = f"terraform import aws_route53_record.{resource_name} {ZONE_ID}_{resource_name}_{resource_type}"
    run(import_command, shell=True, check=True)

3. (Optional) Move Resources in a Module#

In case you are using a Module to manage AWS Route53 resources, you’ll need to move the declaration from resource to module configuration block. This is described more in detail here.

The module declaration/naming would depend on how the module is configured. To demonstrate, the module I use internally requires the name to be of the format resource_name-resource_type. To achieve this, you can call terraform state mv as a subprocess:

# Shells out `terraform state mv` command in the host OS.
def terraform_move(resource_name, resource_type):
    mv_command = f"terraform state mv aws_route53_record.{resource_name} 'module.{MODULE_NAME}.aws_route53_record.route53_record[\"{resource_name}-{resource_type}\"]'"
    run(mv_command, shell=True, check=True)

That’s it! Running terraform plan should now show you the changes and if you imported every record correctly you should not see any drift from the real world state.

You can view the entire script here:

import json
from os import getenv, path
from string import Template
from subprocess import run
from sys import exit


ZONE_ID = getenv("ZONE_ID")
MODULE_NAME = getenv("MODULE_NAME")
ZONE_FILE = getenv("ZONE_FILE")
TERRAFORM_DIR = getenv("TERRAFORM_DIR")

# Returns the variable key if not present in ENV.
def check_env_vars():
    if not ZONE_FILE:
        return "$ZONE_FILE"
    if not ZONE_ID:
        return "$ZONE_ID"
    if not MODULE_NAME:
        return "$MODULE_NAME"
    if not TERRAFORM_DIR:
        return "$TERRAFORM_DIR"
    return ""


# Loads the zone records in a dict
def load_records(zone_file=ZONE_FILE):
    with open(zone_file) as record_file:
        data = json.load(record_file)
    return data


# Writes the dummy Terraform template which is required
# before `terraform import` runs.
def template_dummy_file(resource_name):
    add_dummy_record = Template(
        """
	resource "aws_route53_record" "$resource_name" {
	# (resource arguments)
	}
	"""
    )
    dummy_file_path = path.join(TERRAFORM_DIR, "dummy.tf")
    with open(dummy_file_path, "a") as f:
        f.write(add_dummy_record.substitute(resource_name=resource_name))


# Shells out `terraform import` command in the host OS.
def terraform_import(resource_name, resource_type):
    import_command = f"terraform import aws_route53_record.{resource_name} {ZONE_ID}_{resource_name}_{resource_type}"
    run(import_command, shell=True, check=True)


# Shells out `terraform state mv` command in the host OS.
def terraform_move(resource_name, resource_type):
    mv_command = f"terraform state mv aws_route53_record.{resource_name} 'module.{MODULE_NAME}.aws_route53_record.route53_record[\"{resource_name}-{resource_type}\"]'"
    run(mv_command, shell=True, check=True)


if __name__ == "__main__":
    missing = check_env_vars()
    if missing:
        exit(f"Required env variable {missing} is missing.")
    records = load_records()
    for i in records.get("ResourceRecordSets"):
        resource_name = i.get("Name")
        resource_type = i.get("Type")
        template_dummy_file(resource_name)
        terraform_import(resource_name, resource_type)
        terraform_move(resource_name, resource_type)
        print(f"Imported {resource_name}")

Hope this tiny Python script helps you transition your AWS Route53 records neatly and effortlessly!

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/ripe-atlas-probe-setup/ https://mrkaran.dev/posts/ripe-atlas-probe-setup/ Sat, 03 Oct 2020 02:40:55 GMT Twitter is an amazing thing! For all the shitposting and meme-ing that’s done there, there are some really cool people you get to interact with which wouldn’t have been possible IRL. I happened to stumble upon Swapneel last year in Bangalore at a meetup. He occasionally posts about RIPE Atlas Probes measurements on his Twitter feed and that made me curious and learn more about the Atlas Probe. Call it the Baader-Meinhof effect but just when I was reading his blog post on RIPE Atlas, I found about a Hasgeek workshop which is being conducted by Swapneel himself and well, that’s how I basically got interested in setting up a probe myself.

What is RIPE Atlas Probe#

A Probe is a device used to measure various metrics like DNS, SSL/TLS, NTP, ping, traceroute etc for an upstream target. A network of such probes is useful not just to internet researchers, network engineers, ISP operators, public activists but also to common folks like me who are simply curious about the network/WWW in general. The idea behind this is simple – you get some virtual credits for hosting a probe. You can make use of these credits to run your own measurements to gain insights about the health of your network. The data from these measurements are made publicly available by RIPE NCC and anyone can see these results.

There are 2 ways to host a Probe: Software and Hardware. Since the start of this year, RIPE Atlas is available as software packages for various platforms. Before this, the only way to set up a Probe was to host a Hardware probe by applying for one at RIPE NCC website. These hardware probes are not shipping to India since quite some time as Swapneel mentioned and if you’re in India, your only choice, for now, is to set up a software probe. A few key differences in the 2 kinds of probes:

• Initial Setup: Hardware probe is a plug-n-play device, no setup required. That makes it attractive for a lot of people, to just get a small device and host it.
• Spare Compute: For hosting a probe it is recommended that the probe stays online for as long as possible. This means you need to have a spare compute lying around, like a RaspebrryPi, a VM, a server etc. It’s not really recommended to host a probe on your laptop/desktop - something which has ad-hoc usage pattern. For people who don’t have any spare compute, a hardware probe is more suited for them.
• Tampering Result: With the software probe, there are easy ways of tampering the measurement result, which might give a bad data point to people running the measurements. For eg, a software probe could tamper with the DNS queries for a particular upstream and fool you to believe that there’s a problem with the uplink or the upstream target.

You can read more about Probes here and this FAQ section is highly recommended to clear all the basic doubts you might have about Probes.

Setting up a Software Probe in a Container#

Well, Containers are awesome! It’s easier to do a docker run than figure out the installation instructions for your OS, ensure dependencies are installed and insert 10 other things you have to do. So, I decided to use jamesits/ripe-atlas Docker image to host a probe on my RPi 4.

However, in case you don’t want to use Containers for a reason best known to you, you can visit the official docs and find out instructions for your platform.

You can follow the below steps to set up a probe via Docker:

1. Create a RIPE NCC Account#

You need to register for a RIPE NCC Access Account before proceeding further. Visit the registration page and create an account.

2. Start the container#

docker run --detach --restart=always --log-opt max-size=10m \
        --cap-add=SYS_ADMIN --cap-add=NET_RAW --cap-add=CHOWN \
        --mount type=tmpfs,destination=/var/atlasdata,tmpfs-size=64M \
        -v /var/atlas-probe/etc:/var/atlas-probe/etc \
        -v /var/atlas-probe/status:/var/atlas-probe/status \
        -e RXTXRPT=yes \
        --name ripe-atlas --hostname "$(hostname --fqdn)" \
        jamesits/ripe-atlas:latest

NOTE: In the above case, since I am running the Probe on my RPi which is why I am using latest-armv7l tag for the jamesits/ripe-atlas image. In case you are doing this on an amd64 machine, you should use the latest tag only. You can find more options for the container here.

3. Apply for a Software Probe#

You need to register for a software probe here and fill in the details as mentioned below.

• ASN: You can find the ASN of your public IP by visiting ip2asn.com. You need to enter your public IP here which you can find it by visiting ifconfig.co.

• Public Key: You need to fill the public key generated by your Probe:

cat /var/atlas-probe/etc/probe_key.pub

Note: We are mounting the /var/atlas-probe/etc on the host path inside the container at /var/atlas-probe/etc. This ensures that if you stop/remove the container, your public/private key pair isn’t lost and there’s no need to regenerate/apply for a Probe again. However in case you lose this, the Probe will attempt to generate a new pair and you might need to change the public key at that point of time. So be careful to take a backup of the generated key and store somewhere safe.

After submitting the form, you will receive an email which will have information about your Probe ID.

4. Wait patiently#

After filling the form you need to wait roughly for 15 minutes for your probe to be added to the global network of Atlas Probes. You will receive an email saying your Probe is live now.

Creating Measurements#

It’s pretty simple to create measurements, both via UI and a Python library developed by the RIPE community. However, for the ease and convenience I will demonstrate how to create a traceroute measurement from the UI itself:

Note: You get around 15 Credits/minute for hosting a probe. In case you just deployed a probe you might not have enough credits to perform this measurement. You can come back later to do this when you have sufficient credits.

Visit the dashboard and headover to Measurement tab. On clicking the Create a Measurement button you will be presented a form with various options. Let’s try a Traceroute to mrkaran.dev for this measurement:

To limit the number of probes, I am going to select all Probes in India, by clicking on +New Set -wizard option in the Probe Selection tab.

After submitting the form, you will be presented with a Measurement ID which you can use to download the data at a later point as well.

Visit the Measurement ID page and wait for a couple of minutes for all probes to perform the measurements:

That’s it! You can explore other types of Measurements and different options while performing them to customise your use case.

Summary#

Atlas Probes are pretty easy to host and they are crucial in giving valuable insights about how different clients connect to upstreams and spot issues in Networks. If you can host a probe, you should seriously consider doing that and help increase the number of probes in the network. As of today, the number of probes in India is just around 113 and that is seriously very less.

If you need any help hosting one, don’t hesitate to contact me on my Twitter.

Oh, and for the record, I am hosting 2 Probes currently. Probe 1000991 hosted on my local RPi connected to my home network (ASN24560) and Probe 1001117 hosted on a DigitalOcean droplet in Bengaluru (ASN14061).

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/isp-monitoring/ https://mrkaran.dev/posts/isp-monitoring/ Fri, 18 Sep 2020 02:40:55 GMT I like monitoring stuff. That’s what I do at work and when my home ISP started giving me random problems and I decided it would be nice to monitor my home network as well. There are a couple of ways to go around this, a very popular and OSS solution is SmokePing. SmokePing is written in Perl and is used to visualise network latencies. It’s quite a great solution but for my current stack which involves Prometheus and Grafana, it meant I had to deploy a standalone tool separate from my monitoring stack - something which I wanted to avoid.

So, I looked for other solutions and luckily happened to stumble upon oddtazz in one of the common Telegram groups where he shared his solution for the above: Telegraf ICMP plugin and Grafana. This is exactly what I’ve been looking for but for some reason, I had wrongly assumed Telegraf needs InfluxDB to store the data. Googling a bit more, I found Telegraf supports Prometheus format (amongst a huge list of others!) but this wasn’t so clear in their docs.

I decided to run a Telegraf agent in my RPi connected to my home router over LAN and scrape metrics using Prometheus and visualise graphs in Grafana! For the non-patient readers, here’s what my dashboard looks like!:

Setup#

To get started, we need to download Telegraf and configure the Ping plugin. Telegraf has the concept of Plugins for Input, Output, Aggregating and Processing. What this basically means is that you can configure multiple input plugins like DNS, ICMP, HTTP and export the data of these plugins in a format of your choice with Output plugins. This makes Telegraf extermely extensible, you could write a plugin (in Go) of your choice if you fancy that as well!

Here’s what my telegraf.conf looks like:

# Input plugins

# Ping plugin
[[inputs.ping]]
urls = ["mrkaran.dev", "tailscale.mrkaran.dev", "floyd.mrkaran.dev", "1.1.1.1", "kite.zerodha.com", "google.com", "reddit.com", "twitter.com", "amazon.in", "zerodha.com"]
count = 4
ping_interval = 1.0
timeout = 2.0

# DNS plugin
[[inputs.dns_query]]
  servers = ["100.101.134.59"]
  domains = ["mrkaran.dev", "tailscale.mrkaran.dev", "floyd.mrkaran.dev", "1.1.1.1", "kite.zerodha.com", "google.com", "reddit.com", "twitter.com", "amazon.in", "zerodha.com"]

# Output format plugins
[[outputs.prometheus_client]]
  listen = ":9283"
  metric_version = 2

Firstly, so nice to see an Ops tool not using YAML. Kudos to Telegraf for that. I’d love to see other tools follow suit.

Getting back to the configuration part, input.plugin is a list of plugins that can be configured and I have configured the Ping and DNS plugin in my config. The output is in Prometheus format so it can be scraped and ingested by Prometheus’ time-series DB.

Running Telegraf#

With the above config in place, let’s try running the agent and see what metrics we get. I am using official Docker image to run the agent with the following config:

docker run --name telegraf-agent --restart always -d -p 9283:9283 -v $PWD/telegraf.conf:/etc/telegraf/telegraf.conf:ro telegraf

After running the above command, you should be able to see the metrics at localhost:9283/metrics

$ curl localhost:9283/metrics | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP dns_query_query_time_ms Telegraf collected metric
# TYPE dns_query_query_time_ms untyped
dns_query_query_time_ms{dc="floyd",domain="amazon.in",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 124.096472
dns_query_query_time_ms{dc="floyd",domain="google.com",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 136.793673
dns_query_query_time_ms{dc="floyd",domain="kite.zerodha.com",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 122.780946
dns_query_query_time_ms{dc="floyd",domain="mrkaran.dev",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 137.915851
dns_query_query_time_ms{dc="floyd",domain="twitter.com",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 111.097483

Perfect! Now, we’re all set to configure Prometheus to scrape the metrics from this target. In order to do that you need to add a new Job:

- job_name: "ispmonitor"
  scrape_interval: 60s
  static_configs:
    - targets: ["100.94.241.54:9283"] # RPi telegraf Agent

In the above config, I am plugging my Tailscale IP assigned to my RPi on the port where Telegraf agent is bound to. This is one of the many reasons why Tailscale is so bloody awesome! I can connect different components in my network to each other without setting up any particular firewall rules, exposing ports on a case by case basis.

Sidenote: If you haven’t read Tailscale’s amazing NAT Traversal blog post, do yourself a favour and check it out after you finish reading this one ofcourse!

Anyway, coming back to our Prometheus setup, we can see the metrics being ingested:

Show me the graphs#

Now comes the exciting bit – making pretty graphs. First, let’s discuss what’s the most important data I can extract out of Ping and DNS plugins. These plugins export decent amount of data, but a good rule of thumb while making dashboards is to optimise signal v/s noise ratio. We’ll do that by filtering out only the metrics that we care for.

Let’s checkout all the metrics exported by Ping plugin:

$ curl localhost:9283/metrics | grep ping | grep TYPE
# TYPE ping_average_response_ms untyped
# TYPE ping_maximum_response_ms untyped
# TYPE ping_minimum_response_ms untyped
# TYPE ping_packets_received untyped
# TYPE ping_packets_transmitted untyped
# TYPE ping_percent_packet_loss untyped
# TYPE ping_result_code untyped
# TYPE ping_standard_deviation_ms untyped
# TYPE ping_ttl untyped

Perfect! So, from the above list of metrics, the most important ones for us are:

• ping_average_response_ms: Avg RTT for a packet
• ping_maximum_response_ms: Max RTT for a packet
• ping_percent_packet_loss: % of packets lost on the way

With just the above 3 metrics, we can answer questions like:

• Is my ISP suffering an outage?

If yes, ping_percent_packet_loss should be unusually higher than normal. This usually happens when the ISP has routing is borked and that causes the packet to be routed in a less optimized way and as a side effect packet loss becomes one of the key metrics to measure here.

• Is the upstream down?

If yes, ping_average_response_ms over a recent window should be higher than a window compared to a previous time range when things were fine and dandy. This can either mean 2 things: Either your ISP isn’t routing correctly to the said upstream or the CDN/Region where your upstream is faced an outage. This is quite a handy metric for me to monitor!

How many times have your friends complained “xyz.com isn’t working for me” and when you try to load, it’s fine from your end? There are a lot of actors at play but ping is usually the most simple and quickest way to detect whether an issue persists or not. Of course, this doesn’t work for hosts which block ICMP packets altogether. They are not rare either, like netflix.com and github.com both block ICMP probes for example. For my use case, this wasn’t a major issue as I was able to still probe a decent amount of upstreams all over the world.

With that out of the way, let’s break the dashboard into different components and see what goes behind them.

Ping Response Panel#

To plot this, simply choose a Stat visualisation with the query ping_average_response_ms{url="$url"}. Repeat this panel for the variable $url and you should be able to generate a nice row view like this.

Additonally you can choose Thresholds and the Unit to be displayed in the panel with these options.

Ping Response Time Graph#

The next graph is interesting, it lets me visualise the avg, min, max ping response time as well as the % packet loss plotted on the Y2 (right Y) axis.

Availability Panel#

An interesting query to calculate uptime (just in the context whether the upstream is reachable) is:

100 - avg_over_time(ping_percent_packet_loss[2m])

Since I scrape metrics at an interval of 1m(in order to not ping too frequently and disrupt my actual browsing experience), in this query I am averaging the data points for the metric ping_percent_packet_loss in a [2m] window.

DNS Response Time Graph#

We can similarly query the DNS response time by visualising the average response time for a DNS query. This might be useful only to people self-hosting their DNS servers.

Conclusion#

So with a pretty simple and minimal OSS solution, I was able to setup monitoring for my home network! Over the last few days whenever my ISP had slightest of trouble, I can correlate it with my metrics! I mean I still can’t do anything about it cause the other person on ISP’s customer support is “Did you try rebooting your router” – the quintessential solution to all tech problems. Wish we could reboot this entire damn 2020 as well, but one could hope!

Shoot me for any questions on my Twitter @mrkaran_ :)

Fin!

]]> Karan Sharma https://captnemo.in/blog/2020/09/16/goi-cyberspace/ https://captnemo.in/blog/2020/09/16/goi-cyberspace/ Wed, 16 Sep 2020 00:00:00 GMT I recently did some work on analysing the Indian government cyberspace, thought I should document them somewhere outside of my Twitter¹.

List of GoI websites

I’d made a list of Indian government websites in Jan 2019:

I ran @18F /pulse on Indian Government websites to see how many of them support HTTPS. A quick summary:

Total Websites: 14183
Total Live Websites: 11710 (82%)
Websites with Valid HTTPS: 4753 (40% of all live websites)

Raw Dataset for now: docs.google.com/spreadsheets

— Nemo (@captn3m0) January 15, 2019

The dataset was from 2 sources:

1. GoI Directory
2. crt.sh (All certificates ending in .gov.in were used)

I re-ran the scripts to get an updated list (12842 domains), then tabulated them against the public-suffix² for each. There is a long-tail, and I’ve published results here. Here are the top public suffixes for Indian government sites:

Public Suffix	Domains
nic.in	2454
gov.in	7259
in	528
ac.in	490
com	568
co.in	171
org.in	168
edu.in	117
org	844
res.in	134
net.in	12
net	38

Sanskari Proxy

This was a long standing idea on my ideas repo:

A lot of Indian Government websites are inaccessible on the public internet, because they geo-fence it to within Indian Boundaries. The idea is to make a Indian Proxy service that specifically works only for the Geo-fenced Indian government websites.

For eg, if uidai.gov.in is inaccessible, hitting uidai.gov.sanskariproxy.in will get you the same result, proxied via our servers.

Since I’d made an updated list of GoI websites, this seemed easy enough. I realized that setting up uidai.gov.sanskariproxy.in would likely count as impersonation under the Indian law, so I did the next best thing: run an actual proxy. Here’s the announcement tweet:

Are you a security researcher outside India? Do you hate getting geoblocked to Indian government websites?

Well, I made a proxy for security researchers outside India to access Indian government websites without resorting to shady VPNs.
captn3m0/sanskari-proxy

— Nemo (@captn3m0) September 5, 2020

Project page is https://github.com/captn3m0/sanskari-proxy, and if you’d like to get access - please reach out.

Cyberspace Ownership

I’d planned to get a complete list of geoblocked websites next. While I’m progressing on this front, the results have been inconsistent/inaccurate so far. As an intermediate step, I’d made a list of IPs against every domain³, which looked like this:

Domain	IP Address
aavin.tn.gov.in	164.100.134.148
abnhpm.gov.in	14.143.233.34
agnii.gov.in	13.232.216.65
ap.gov.in	117.254.92.53
aponline.gov.in	125.16.9.130
appolice.gov.in	118.185.110.147
attendance.gov.in	164.100.166.114
cgg.gov.in	112.133.222.115

While running numerous nmap scans (and failing), I start checking the ASN⁴ for some of these IPs to see who was hosting each website - especially the ones I was finding were blocked.

I stumbled upon a bulk IP to ASN service by Cymru, ran all the IPs against it and published the results. Here’s the important graph:

As you can expect, NIC⁵ has the highest share, with NKN⁶, BSNL, and CtrlS following at roughly 5% each. There are a few other chart on the twitter thread, and the raw data is available here with interactive versions of each visualization.

What next?

I’m working on running and comparing connectivity scans to these IPs to get a better understanding of the geoblocking situation. There’s also some issues with the domain list, as it seems to be missing lots of domains - so more corrections are needed.

1. Twitter decided to suspend 12 different accounts I had access to recently - I’m starting to get wary of using Twitter for archival now. ↩

2. A “public suffix” is one under which Internet users can (or historically could) directly register names. For eg - nic.in or github.io. Mozilla manages the list at https://publicsuffix.org/. ↩

3. There are issues with this approach, since domains do resolve to multiple IPs. But this is okay for the rudimentary analysis I’ve been doing so far. ↩

4. Autonomous Systems (AS) is how the internet is sliced up and managed by different entities. Each AS (usually an ISP) is responsible for routing within its network, while announcing network routes on how it can be reached. ↩

5. The primary government office (under MeitY) that provides infrastructure and support for government IT services. ↩

6. National Knowledge Network is a multi-gigabit research and education network that provides a high speed network backbone for educational institutions in India. ↩

]]> Nemo https://ibcomputing.com/setting-up-whatsapp-telegram-bridge-using-matterbridge-2020/ https://ibcomputing.com/setting-up-whatsapp-telegram-bridge-using-matterbridge-2020/ Fri, 11 Sep 2020 14:12:49 GMT Two years ago I wrote an article about bridging WhatsApp and Telegram groups together. But that solution isn’t working anymore. So I figured out another method using WhatsApp web. The brdiging software we are going to use for this is called matterbridge. the software we use is matterbrige. This bridge is not only for Whatsapp and Telegram but for So many messaging protocols.

Natively supported apps

• Discord
• Gitter
• IRC
• Keybase
• Matrix
• Mattermost 4.x, 5.x
• Microsoft Teams
• Nextcloud Talk
• Rocket.chat
• Slack
• Ssh-chat
• Steam
• Telegram
• Twitch
• WhatsApp
• XMPP
• Zulip

The whatsapp linking part is based on go-whatsapp which is a Go version of  whatsapp-web-reveng.

I think that’s enough for the intro, let’s get back to the main point

Install and Run

Download the latest version from here
select the required package. im using linux for this tutorial so i select matterbridge-1.18.3-linux-64bit (this version will change)
then move to the Configuration part

Step 1

Create an empty matterbridge.toml file inside a folder contains downloaded matterbridge binary file.

Step 2

Copy the following contents to the newly created matterbridge.toml file:

[telegram.mytelegram]
#See https://core.telegram.org/bots#6-botfather 
#and https://www.linkedin.com/pulse/telegram-bots-beginners-marco-frau
Token="Yourtokenhere"
RemoteNickFormat="({PROTOCOL}) <{NICK}> "
MessageFormat="HTMLNick"

[whatsapp.mywhatsapp]
# Number you will use as a relay bot. Tip: Get some disposable sim card, don't rely on 
# your own number.
Number="+48111222333"

# First time that you login you will need to scan QR code, then credentials will be saved in 
# a session file. 
# If you won't set SessionFile then you will need to scan QR code on every restart.
# optional (by default the session is stored only in memory, till restarting matterbridge)
SessionFile="session-48111222333.gob"

# If your terminal is white we need to invert QR code in order for it to be scanned properly
# optional (default false)
QrOnWhiteTerminal=false

# Messages will be seen by other WhatsApp contacts as coming from the bridge. 
# Original nick will be part of the message.
RemoteNickFormat="[{PROTOCOL}] @{NICK}: "

Step 3

Change the Token and Server keys above to yours. For more details you may watch this video.

In Telegram, go to botfather and create a bot and paste the token as Token="Yourtokenhere"
Enter your whatsapp number as Number="+48111222333"

Step 4

Add the gateway configuration to your config file.

[[gateway]]
name="gateway1"
enable=true

[[gateway.inout]]
account="whatsapp.mywhatsapp"
channel="xxxxxx@g.us"

[[gateway.inout]]
account="telegram.mytelegram"
channel="-xxxxx"

Step 5

Now your configuration file should look like this:

[telegram.mytelegram]
#See https://core.telegram.org/bots#6-botfather 
#and https://www.linkedin.com/pulse/telegram-bots-beginners-marco-frau
Token="1279548291:AAFg3s7F-JQlQQlucSBf3PGwyOss2Vog0J4"
RemoteNickFormat="({PROTOCOL}) <{NICK}> "
MessageFormat="HTMLNick"

[whatsapp.mywhatsapp]
# Number you will use as a relay bot. Tip: Get some disposable sim card, don't rely on 
# your own number.
Number="+48111222333"

# First time that you login you will need to scan QR code, then credentials will be saved in 
# a session file. 
# If you won't set SessionFile then you will need to scan QR code on every restart.
# optional (by default the session is stored only in memory, till restarting matterbridge)
SessionFile="session-48111222333.gob"

# If your terminal is white we need to invert QR code in order for it to be scanned properly
# optional (default false)
QrOnWhiteTerminal=false

# Messages will be seen by other WhatsApp contacts as coming from the bridge. 
# Original nick will be part of the message.
RemoteNickFormat="[{PROTOCOL}] @{NICK}: "

[[gateway]]
name="gateway1"
enable=true

[[gateway.inout]]
account="whatsapp.mywhatsapp"
channel="xxx-xxx.@g.us"

[[gateway.inout]]
account="telegram.mytelegram"
channel="-xxxx"

Step 6

Find channel ID for Telegram and WhatsApp. Finding Telegram group ID is relatively easy.  You can use the Telegram bot API to get the group ID. But I’m taking a simple shortcut here.  Adding another bot to retrieve group ID, which is @tchncs_bot. This is a bot that bridges Matrix and Telegram groups. Just add the bot and sending the message /id (including the forward slash, which makes the message a command) will return the group ID.

Unlike in Telegram’s case, finding channel name in WhatsApp isn’t that easy. I couldn’t find any easier method from matterbridge documentation as well. So enter your WhatsApp group name there and run the bridge (Warning: you will definitely get an error!)

Step 7

Run it with the following command (change the version number if you are using later version)
./matterbridge-1.18.3-linux-64bit -conf matterbridge.toml

Now you can see a QR code in terminal. If it is too big that you weren’t able to scan it, then zoom in the terminal and try again. Then scan the QR code with WhatsApp web option from your mobile phone. Now your WhatsApp web is connected with the matterbridge. Next step is to identify the channel name for WhatsApp. By checking the terminal output you can see your group name and a number that ends with @g.us. Copy that and paste it in the channel part of WhatsApp and run again. Now your bridge is ready and it will start working. You can also share image, files and audios using this bridge. If you want to bridge multiple groups the gateway part can be changed to

[[gateway]]
name="gateway1"
enable=true

[[gateway.inout]]
account="whatsapp.mywhatsapp"
channel="xxxx-xxxx@g.us"

[[gateway.inout]]
account="telegram.mytelegram"
channel="-xxxx"

[[gateway]]
name="gateway2"
enable=true

[[gateway.inout]]
account="whatsapp.mywhatsapp"
channel="xxx-xxxx@g.us"

[[gateway.inout]]
account="telegram.mytelegram"
channel="-xxxx"

 

Drawbacks

This brdige reies on WhatsApp web. So if you have to make this solution work, you should be online on both mobile as well as on WhatsApp web all the time. Other solution you could try is to run any emulator on server. Also, the matterbridge should be running all the time. So you have to choose a server for that. Cloudcone provides low cost servers with good quality. You can use this link to using cloudcone.

The post Setting up WhatsApp Telegram Bridge Using Matterbridge 2020 appeared first on IB Computing.

]]> Mujeeb cpy Android Telegram Tutorials https://captnemo.in/monopoly-deal/ https://captnemo.in/monopoly-deal/ Wed, 24 Jun 2020 00:00:00 GMT 1 set. But if you use it to win the game, the game ends immediately. Hence, Deal Breaker will always end up being the last card of the game. If you are playing a game with the Deal Breaker card, you’d want to save it till the very end, and win the game with it. The only possible case for not winning is the other player having a “Just Say No” card, and playing it on the Deal Breaker to negate your move. Ergo, the metagame converges to the following: Any game with Deal Breaker will end up having the Deal Breaker as the last turn. The only way to prevent someone else from winning with the Deal Breaker is to play a Just Say No on the Deal Breaker. If you have a Just Say No card, you must save it till the end of the game for the Deal Breaker. There are 2 Deal Breakers, and 3 Just Say Nos in the game. However, considering a single Deal Breaker is enough to win the game, and the chances of you getting a second of either card are fairly small - both the Deal Breaker and Just Say No cards will end up getting hoarded for the endgame. What this results in is something that breaks the fundamental rule of game design: Players are disincentivized from playing the Just Say No card. As any Exploding Kittens player can confirm, playing a Just Say No card is one of the coolest moves in the game. It lets you stick it to the player who dares ask you for 8M3 rent. It lets you pretend you’re counting your money, and then pull out a trump card and feel awesome! By disincentivizing players from playing the coolest card in the game, the Deal Breaker card makes things less fun. And that breaks our “rule of fun”. In fact, the mere existence of a Deal Breaker card changes the equation. Note that there may be cases where you lose with a Just Say No card because you were hoarding it for the eventual Deal Breaker (which might never come). Someone asks you for 4M rent, and you have to pay up despite having a Just Say No card, because you must save the damn card for when someone steals your set. There are a few rare exceptions, but the Deal Breaker creates too many plays where not playing the Just Say No is indeed the correct move. So here is the more interesting corollary observation: The mere existence of the Deal Breaker card breaks the game by making the Just Say No card unplayable and worthless.4 Hence, if you’re playing Monopoly Deal, please house-rule the Deal Breaker card and make it easier for everyone. Two easy ways are: Remove the card from the game entirely. Reduce the power of the game to be same as a Forced Deal, except let it break a set. I have the slides here, but they don’t stand well on their own. ↩ Any game that eliminates players from the game breaks this rule. Popular examples are Monopoly and Mafia/Werewolf. Also see this amazing post on the biggest mistake that Guillotine makes (The “Callous Guard” card). ↩ Turns out, that there is no symbol in the unicode for Monopoly Money ↩ We decided Deal Breaker might make sense in 6+ player games with many more cards, where it might help even the playing grounds a bit for a losing player (much more likely) instead of helping the almost-winning-player score a victory. ↩]]> I have a presentation I sometimes give about Monopoly being a terrible game¹. I usually end it by pointing the audience to Monopoly Deal, which I introduce it as “the only Monopoly edition you can enjoy”.

The advantages are obvious:

1. Much shorter game length.
2. No dice rolls.

Even if you lose badly on luck, you’ve only lost 15 minutes. While it is better than Monopoly, that isn’t to say it is a well designed game (6.3 on BoardGameGeek).

The one major dent in the otherwise decent game is the “Deal Breaker” card, which breaks the game. I’ve house-ruled the game since forever to keep the card out of the game, since it breaks the most important rule of game design:

Games should be fun²

Deal Breaker stops people from doing that. How? Read on.

---

A lot of metagaming discussion with friends resulted in the following observations:

• The Deal Breaker is a very powerful card (It takes you 1/3rd of the way to a victory).
• You can assume the single Deal Breaker card to be worth a complete set.
• The best use of such a card is to win the game. Using it earlier means giving other players a chance to drag you down from 2->1 set. But if you use it to win the game, the game ends immediately.
• Hence, Deal Breaker will always end up being the last card of the game.

If you are playing a game with the Deal Breaker card, you’d want to save it till the very end, and win the game with it. The only possible case for not winning is the other player having a “Just Say No” card, and playing it on the Deal Breaker to negate your move.

Ergo, the metagame converges to the following:

1. Any game with Deal Breaker will end up having the Deal Breaker as the last turn.
2. The only way to prevent someone else from winning with the Deal Breaker is to play a Just Say No on the Deal Breaker.
3. If you have a Just Say No card, you must save it till the end of the game for the Deal Breaker.

There are 2 Deal Breakers, and 3 Just Say Nos in the game. However, considering a single Deal Breaker is enough to win the game, and the chances of you getting a second of either card are fairly small - both the Deal Breaker and Just Say No cards will end up getting hoarded for the endgame.

What this results in is something that breaks the fundamental rule of game design:

Players are disincentivized from playing the Just Say No card.

As any Exploding Kittens player can confirm, playing a Just Say No card is one of the coolest moves in the game. It lets you stick it to the player who dares ask you for 8M³ rent. It lets you pretend you’re counting your money, and then pull out a trump card and feel awesome! By disincentivizing players from playing the coolest card in the game, the Deal Breaker card makes things less fun. And that breaks our “rule of fun”.

In fact, the mere existence of a Deal Breaker card changes the equation. Note that there may be cases where you lose with a Just Say No card because you were hoarding it for the eventual Deal Breaker (which might never come). Someone asks you for 4M rent, and you have to pay up despite having a Just Say No card, because you must save the damn card for when someone steals your set. There are a few rare exceptions, but the Deal Breaker creates too many plays where not playing the Just Say No is indeed the correct move.

So here is the more interesting corollary observation:

The mere existence of the Deal Breaker card breaks the game by making the Just Say No card unplayable and worthless.⁴

Hence, if you’re playing Monopoly Deal, please house-rule the Deal Breaker card and make it easier for everyone. Two easy ways are:

1. Remove the card from the game entirely.
2. Reduce the power of the game to be same as a Forced Deal, except let it break a set.

1. I have the slides here, but they don’t stand well on their own. ↩

2. Any game that eliminates players from the game breaks this rule. Popular examples are Monopoly and Mafia/Werewolf. Also see this amazing post on the biggest mistake that Guillotine makes (The “Callous Guard” card). ↩

3. Turns out, that there is no symbol in the unicode for Monopoly Money ↩

4. We decided Deal Breaker might make sense in 6+ player games with many more cards, where it might help even the playing grounds a bit for a losing player (much more likely) instead of helping the almost-winning-player score a victory. ↩

]]> Nemo https://captnemo.in/one-card-to-rule-them-all/ https://captnemo.in/one-card-to-rule-them-all/ Mon, 22 Jun 2020 00:00:00 GMT A lot of companies have come up with the idea for reducing all your cards into a single piece of plastic. Here’s a summary of all the ones I could find, and their fate. Beware: this field is very much a startup graveyard. The only remaining survivor seems to be Curve¹ but it’s also the first one that’s attempting this outside of US.

There seem to be a lot of challenges (regulatory, financial, and technical) before such a thing becomes reality. And there’s Apple/Samsung Pay as well. Here’s a summary of all the companies I could find in this space. If I’ve missed any, please let me know, and I’ll add them here.

Curve (2015-)

Curve allows you to spend from any of your accounts using just one card, clearing the clutter from your wallet and simplifying your finances.

• Curve has raised a total of $74M at a valuation north of $250M, out of which £6M were from a crowdfunding campaign in 2019.
• Their waitlist is at 800,000+ users.
• Curve faced flak for failing to disclose its usage numbers (<100,000 monthly active users out of total 500,000) to crowdfund investors.
• Only works with Visa/MasterCard. Used to work with Amex, but Curve has a history of working and being blocked by Amex repeatedly.
• See this page for some details on how it works.

Coin (YC W13) (2012-2017)

• Coin raised a total of $15.6M.
• Shipped in 2015 April.
• Acquired by Fitbit in May 2016.
• Shutdown by Fitbit in Feb 2017.
• Related: Google acquired FitBit in 2019 Nov.

Plastc (2014-2017)

• Single dynamic card with e-Ink touchscreen. See obligatory launch video with fancy AR.
• Crowdfunded $9M on pre-orders in October 2014.
• Delayed launch to April 2016, and then again to September 2016.
• Shut down and declared bankruptcy in April 2017.
• Complete story via digg.com.

Stratos (2015-2015)

• Was supposed to cost $95/year.
• Announced May 2015.
• Raised $6.63 million over three rounds of financing.
• Ran out of money by December 2015.
• Sold to Ciright One to avoid collapse.
• Seems to be dead now.

Swyp (2014-2017)

• Had a pre-order campaign in 2014.
• Raised $5M from Khosla Ventures in 2017.
• Tried to pivot in 2017, failed.
• Offered customers a debit card with an app called Hoot in 2017 as an alternative to refunds. I don’t think the Hoot card ever materialized.
• Tilt (Swyp’s payment processor) also ceased operations in 2017, due to unrelated reasons.
• Swyp finally shut down in December 2017.

Final (YC W15) (2014-2017)

• Final was with a credit card with a different number for every website. As a independent card, Final doesn’t exactly fit in this list, but it’s a very relevant and loved product.
• Announced itself with a snazzy video in mid 2014
• Raised $4M and launched in August 2016, but remained invite-only till the very end.
• Final’s blog has some interesting content: A Request for Credit Cards program to build card-issuance backed businesses and the Payment card landscape for 2017.
• Shut down in december 2017 and acquired by Goldman Sachs in an acqui-hire.

Indian Landscape

India hasn’t seen a true single-card app yet, but there have been lots of related attempts:

• Infino promises a single-card, but it hasn’t launched yet. It had a public roadmap, but never launched and ultimately pivoted to a Coupon Aggregator called Meet Donut, which shut down.
• IndusInd bank has tried a dual-chip debit+credit card. Union Bank has a similar card.
• IndusInd has also tried an interactive credit card with 3 buttons.
• FamPay, Fold, Niyo, OneCard, Donut🪦, vCard🪦, Slice ² and lots of other startups are doing co-branded issuance, but that’s not the same thing.

With the emergence of UPI, and low penetration rate of credit cards, I don’t see a market in India - but I’d love to be proven wrong.

---

Did I miss anything? Reach out and let me know.

Thanks to Harman for reviewing drafts of this, and PRL for getting me interested enough to document this.

1. Curve used to live on imaginecurve.com, then switched to curve.app and now to curve.com, which must have cost them millions. ↩

2. Unlike others on the list, vCard is entirely a virtual card, and supports UPI transfers from your credit limit. ↩

]]> Nemo https://captnemo.in/blog/2020/06/07/his-dark-materials/ https://captnemo.in/blog/2020/06/07/his-dark-materials/ Sun, 07 Jun 2020 00:00:00 GMT A long time ago, I tried to do a readthrough for Game of Thrones (Book 1) alongside the first season. I managed to reach Episode 5, before I sped through the rest of the book, but I tried.

Trying something similar for His Dark Materials, which is a great series if you’re looking to watch something new. Instead of noting down Chapter/Book equivalence (like I tried last time), going to write down my thoughts here as I’m reading along. Spoiler Warning for the entire first season obviously.

Overall

The show is very tightly knit with the book with a few over-arching changes from the story-telling perspective:

• You get to see other points-of-view, other than just Lyra. Helps establish what else is happening, especially in the other worlds.
• A lot of infodumps are prevented, or better, broken down into multiple sessions.
• The major change from the first book is ofcourse showing Will’s PoV and our earth.

Chapter 1

• The opening scene with the great flood sets some context, but isn’t in the books.
• The Master/Butler chat on the wine poisoining happens much later in the books.
• There is a lot of foreshadowing around Lyra’s parentage that happens in the first 2 chapters that is entirely missed in the show.

Chapter 2

• The entire Grumman’s skull and hunt sub-plot hasn’t shown up in the book so far (presumably because we only see Lyra’s PoV).
• The party scene is very-well handled (with all the subtle changes for the better. Superb acting as well :)
• The hiding-lyra-in-the-boat scene is merely given a passing mention in the book, but so well done in the show.

Chapter 3

• Splitting Lyra’s parentage reveal (Coulter reveals her father) is a smart move in the show.
• The show changes Lyra’s kidnappers from Turkish slavers to Gobblers.
• The Alethiometer reveal happens with both Father Coram and John Fa in the book. The section also has a huge infodump, especially since it involves the parentage reveal. The show breaks it into 3: alethiometer reveal with Father Coram, a previous interrogation of Lyra with John Fa, and Lyra’s parentage reveal (mother) with Ma Costa.

Chapter 4

• The one notable “not-in-the-book” scene is the Coulter’s meeting with Iofur.

Chapter 5

• Interesting to note that the characters of Billy and Roger are fused in both the film and the TV adaptations.

Chapter 6

• Lyra starts a fire in the books, but the show makes it more dramatic by destroying the machine.
• The balloon ride covers a lot more in the books.

Learnings

Overall, the show has been nicely adapted so far, and I think there’s a few reasons:

1. The show barely messes with Lyra’s timeline. Important to ensure this to avoid creating cascading issues down the path.
2. Majority of the changes are either made on kill-able subplots, or side-plots that show us what’s going on elsewhere.
3. And finally, the show spends time on where the medium works best. The scaring scene in Chapter 2, for eg.

I’m still sad that the ghosts in the crypt don’t get to be seen, though.

]]> Nemo https://www.prashanthudupa.com/aarohi-open-learning/ https://www.prashanthudupa.com/aarohi-open-learning/ Thu, 04 Jun 2020 03:58:01 GMT Prashanth Udupa Unschooling https://www.prashanthudupa.com/my-journey-with-scrite/ https://www.prashanthudupa.com/my-journey-with-scrite/ Thu, 21 May 2020 07:45:54 GMT Prashanth Udupa Moments Screenplays Technical https://captnemo.in/starwars-beskar-viewing-order/ https://captnemo.in/starwars-beskar-viewing-order/ Mon, 04 May 2020 00:00:00 GMT I tweeted out my recommended viewing order for Star Wars recently¹:

Happy #StarWarsDay folks. If you've somehow managed to avoid watching Star Wars, here's my recommended viewing order (No Spoilers, 1/n) pic.twitter.com/K70O4ydCB7

— Nemo (@captn3m0) May 4, 2020

Thought I should expand a bit on the what and why. Spoilers towards the end (marked). I also went ahead and named it:

The Beskar Order

1. Rogue One: A Star Wars Story
2. Star Wars: Episode IV - A New Hope.
3. Star Wars: Episode V - The Empire Returns
4. Star Wars: Episode VI - Return of the Jedi
5. The Mandalorian (S01E01-04)
6. Star Wars: Episode VII - The Force Awakens
7. The Mandalorian (S01E05-08)

---

If you’ve enjoyed the above, you should pick between the following 2 next:

1. If you liked the core Skywalker Saga (Episodes IV-VI), and want to explore Anakin’s origins - Go and watch the prequel trilogy (Episode I, II, III).
2. If you liked Force Awakens, you should go finish the sequels (Star Wars Episode VIII, IX)

In either case, I want to leave this as a choice to the viewer. The prequel trilogy has a lot of flaws. The Machete order famously drops an entire film, and it wasn’t even trying to make room. Go to the prequels if you want to explore the lore. On the flip side, if you liked how Disney handled Episode VII, and want to see closing arcs for the major characters, try the sequels. I wouldn’t recommend interleaving them - it doesn’t get you much and makes it confusing.

If you’re still here after finishing both of these (that makes for a total of 10 films and 1 season of telly) - you ought to explore for yourselves. Here’s suggestions depending on what you’d like:

Clone Wars (TV Series, 7 seasons)

for exploring the franchise at a less grand scale. There’s an Essential viewing order, which covers all the major arcs and best episodes.

The Mandalorian (Season 2, Oct 2020)

To find out what happens to Baby Yoda

Star Wars: Rebels (TV Series, 4 Seasons)

If you want to explore new characters and like something Firefly-esque.

There are boardgames, RPGs, and some really great books in the franchise as well. Pick what you’d like to explore.

---

Inspiration

The classic Machete Order which does a lot of great things, by skipping a film, preserving tension and plot-twists. Also of note are the various fan-edits, of which I’ve only ever tried The Phantom Edit.

Rationale

I tried to optimize for a few things:

1. Fun while watching the series. So good stuff comes first, paired films etc, and intentionally including The Mandalorian.
2. Easy stoppage. In case you don’t like the series, you should be able to stop midway, and still have seen the important/best bits.
3. Sticking to chronological order in the stuff I picked (as much as possible). Sticking to chronology makes it easier to consume.
4. Total time. I don’t want to prescribe a “complete-viewing-order”, but rather a “starting point”.

(1) is easy to optimize for. (2/3) results in things getting thrown around a bit, and (4) means I leave out stuff that you should pick for later.

Why not include __?

This is not meant to be an exhaustive order, and I was optimizing for total time. Important mentions:

Solo

Not really essential viewing.

Star Wars: The Clone Wars

The film is terrible (5.9 on IMDB), because it wasn’t meant to be one.

Star Wars: The Clone Wars TV Series

I wanted to stick to the main saga, and its just too damn long to recommend casually in any viewing order.

Star Wars: Holiday Special

I still can’t bring myself to finish it. It isn’t even canon any more (Life Day is)

Legends/Resistance

Again, not essential viewing.

FAQ

Who is this for?

Recommended for first-time viewers. If you are doing a rewatch, I recommend following The Beskar Machete order.

Have you watched everything Star Wars?

I can’t even claim to have watched all 12 films, because I couldn’t finish The Clone Wars (movie). I’m still watching Clone Wars (TV series).

Why Beskar?

I wanted something that would work well with Machete, for the hybrid order. It also makes a point about The Mandalorian ² belonging in the order.

I don’t have this much time!

I’ve tried to optimize for viewing time already. If you wanna trim further - you’ll be left with just the original trilogy (Episode IV, V, VI). Alternatively, just watch The Mandalorian - it stands very well by itself.

Did you backdate this post while publishing?

Yes. I wrote it just a few days after May 4th, and thought it would be nice.

---

More Rationale (SPOILERS AHEAD)

SPOILERS FROM HERE ONWARDS FOR ALL 12 FILMS. READ AT YOUR OWN PERIL

Why start with Rogue One?

Rogue One is a great film, and I love how well it segues into A New Hope. Watching them both back-to-back makes for a great experience. You have this ragged group that has laid down their lives for just a memory chip - and you get to see that bloom into an entire saga. Finishing the original trilogy from there makes sense. The Machete order strongly recommends against starting with Rogue One, but I’ve tried it and it works.

Why not stick with the Machete order as well?

The Machete order goes (IV, V, II, III, VI), deciding to leave out Episode I, and wedging Episodes II, III before you see Return of the Jedi. I was optimizing for time here a bit, and I had to leave the prequels as “for-later” in order to make space for the remaining. If you’re doing a rewatch, and aren’t short on time - you can totally follow it. This is what it morphs into:³

Beskar Machete Order

• Rogue One: A Star Wars Story
• Star Wars: Episode IV - A New Hope
• Star Wars: Episode V - The Empire Returns
• Star Wars: Episiode II - Attack of the Clones
• Star Wars: Episode III - Revenge of the Sith
• Star Wars: Episode VI - Return of the Jedi
• The Mandalorian (S01E01-04)
• Star Wars: Episode VII - The Force Awakens
• The Mandalorian (S01E05-08)
• Star Wars: Episode VIII - The Last Jedi
• Star Wars: Episode IX - The Rise of Skywalker

Why add the Mandalorian at all?

Because frankly - it is both a piece of art, and the best entry into the Star Wars canon in a long time. It also fits into chronological order just after Return of the Jedi - you see how the New Republic has been incompetent, and the ashes of the Empire. You get to experience the power vaccuum in the galaxy, which hopefully makes sense before jumping into The Force Awakens and rise of the First Order.

Why jump to Episode VII (The Force Awakens) instead of finishing The Mandalorian?

We jump a bit ahead (before finishing The Mandalorian) to “A Force Awakens”, getting to just the start of Rey’s story. This is the only “chronology break” in the order, but has no side-effects⁴. The reason for the jump (as opposed to finishing The Mandalorian first) is to have a switch in pace. While I love The Mandalorian, I think pacing it out makes it better.

Why keep Episode VII (The Force Awakens) but not the other sequels?

The best and the worst thing about The Force Awakens is that it is very much “Star Wars”. It doesn’t take any risks, sticks to the tropes, and more importantly - it closes mostly as a self-contained film. Yes, there are a few plot-hooks (Rey’s parentage, Luke, Finn’s coma), but given how badly they are resolved in the following films - it seems Disney didn’t have any better idea to the answers than the viewers. It also gives you a “tasting experience” of the sequels. The sequels have always been polarizing, and watching it gives you a better heading to make the choice b/w Prequels/Sequels later on in the order.

We close with The Mandalorian finale. The Mandalorian isn’t, strictly speaking, essential viewing. While there are hooks, it doesn’t really change anything of consequence to the main saga (at least not in Season 1). But frankly, it is so well made - you deserve to enjoy it. Just look at the trailer:

If you have feedback, send me a tweet. If you’re reading this in the future, note that this was written in May 2020 and could not include media yet to be published.

1. Happy Star Wars Day! ↩

2. Beskar is the Star Wars universe’s Vibranium, and features majorly in The Mandalorian as a minor plot device. ↩

3. The arguments against starting with Rogue One don’t even apply to rewatches, so we ignore the Machete Guideline to keep it to the end. ↩

4. In other words, watching The Force Awakens can’t alter the experience of watching the last few episodes of The Mandalorian season 1. ↩

]]> Nemo https://mrkaran.dev/posts/home-server-updates/ https://mrkaran.dev/posts/home-server-updates/ Thu, 23 Apr 2020 02:40:55 GMT 100.97.222.106:51516: read: connection reset by peer Through some trial and error and reading the docs, I figured that flannel is running as a CNI in k3s. Now the problem is flannel itself is an overlay network. But… Tailscale already is an overlay network (Wireguard) so the packets are not being routed correctly and being dropped halfway in the master node (I am guessing the DNAT/SNAT translation botched up here). The trick was to just change flannel backend to run in the host namespace only. That solved the above issue for me. However, I still had one more issue. The DO node’s public IP was being advertised, while the agent was running on Tailscale network interface so the master was never able to reach the agent. Similarly, when the agent tried to communicate with the server, the private IP of the node was being advertised. Setting the --node-external-ip in k3s config seemed to have fixed the problem. Now all of the nodes in the cluster had proper Tailscale IPs advertised and the node went to Ready state, at last! Who let the DNS out# So, I’ve a chicken and egg problem in my setup. Since my laptop runs a Tailscale agent and whenever I boot up my systemm, Tailscale attempts to posts logs to log.tailsclae.io and fails to start if it cannot. The problem here is who resolves the DNS for me? I run a local DNS server with CoreDNS forwarding my queries to Adguard. Now if I can’t reach Adguard (since Tailscale agent hasn’t initialised), how am I supposed to resolve log.tailscale.io? I did what any sane guy would do, write a simple hacky bash script: #!/bin/bash sudo chattr -i /etc/resolv.conf sudo echo 'nameserver 1.1.1.1' > /etc/resolv.conf echo "changed dns server to 1.1.1.1" sudo tailscale up sudo echo 'nameserver 127.0.0.1' > /etc/resolv.conf echo "changed dns server back to 127.0.0.1" sudo chattr +i /etc/resolv.conf Yes, it’s quite insane. Also, I’ve not been able to figure out how to stop NetworkManager from changing my /etc/resolv.conf so I rely on a hack (documented in Arch official docs), to lock the file so any process cannot modify it. Quirky, but works! Storage# Unfortunately, I don’t have any external HDD/SSD so I am postponing running any stateful workloads till I get one soon (whenever lockdown gets over in my area). I plan to deploy an NFS server so I can run stateful workloads across any node and have redundancy in form of cloud backups. I’ve also heard cool things about Longhorn but unfortunately, it doesn’t have ARM support. Final Thoughts# Well, I am quite stoked by my current setup right now. Learnt a bunch of cool things around NAT traversal, zero-trust networking, realised the old days of LAN were so much better (not that I am that old to have experienced, my first internet connection was rather broadband at home, not even dial-up). Tailscale opens up a lot of new opportunities for corporate VPNs and it is something definitely to watch out for as they continue improving the product. Also, I was super elated when bradfitz himself had commented on my tweet, not gonna lie! {{< tweet 1249552822674149376 >}} Credits# First, a major thanks to my friend sarat with whom I pair debugged a lot of the above issues and since he also runs a home server, he was my go-to person to ask doubts! Here are some links that I collected while setting all of this up and might be useful as references if you’re planning a similar setup! How Tailscale Works What are these 100.x.y.z addresses? k3s config reference P2P across NAT Remembering the LAN Cloudskew (wonderful tool for creating architecture diagrams) If you’ve any questions please find me on Twitter @mrkaran. You can find all of the setup on GitHub. Till next time, if we survive the pandemic!]]> For those of you who are new to my blog, I had written about my home server setup (hydra) a few months back. Since then I’ve tweaked the setup a bit and made some changes to how I organise/deploy applications inside my cluster. I’ll be talking about the updates made so far and the reason behind them.

A brief overview of what has changed from hydra v0.1 to hydra v0.2:

• Replaced Wireguard with Tailscale

• Added a new worker node (residing in DigitalOcean, Bengaluru) to the existing RPi k3s cluster.

• Shifted from PiHole to Adguard DNS upstreaming to Unbound.

• Containerised all workloads and deployed on K8s (using k3s platform).

The setup looks something like this now:

Let’s take a quick look at some of the above things in detail!

Using Tailscale#

So, before we jump to why I started using Tailscale, let’s address a few things about Wireguard which bothered me.

• Adding new devices is a real PITA and quite often laziness kicks in to generate a QR code or Private/Public key pair. So, for those of you unaware, Wireguard needs a client config to add the server endpoint and it’s public key for the client to communicate with the server. You need to add the client’s private key on the server-side as well for the exchange of encrypted packets to happen. Doing all of this manually has been one of the reasons I’ve not added/updated my devices as regularly as I’d like to do. For eg, on my recently bought iPad, I just haven’t bothered to do all of this, cause ugh I am lazy?

• Having a central VPN server to talk to my RPi from my local network just doesn’t seem right you know? Especially when both devices are literally lying in the same room, sending the packets to a server somewhere in Electronic City (DO, blr region) from JP Nagar (where I live) feels like totally unnecessary. I really needed a mesh network to reduce wasted bandwidth and latencies. Having a central VPN server is also a SPOF, not that my home-server runs any mission-critical workloads, but still good to avoid where we can. Where else can I flex my ops-chops skills if not my home server, eh?

I started looking at different mesh VPN setups and Tailscale attracted me the most. I heard about Tailscale first time when I saw bradfitz’s post about leaving Google and joining Tailscale on HN. Interestingly, Tailscale is built on top of wireguard-go (the userspace implementation of Wireguard kernel module). Since I was already familiar with Wireguard and had been using it for almost a year, I got curious on Tailscale.

Tailscale basically sets up a point-to-point encrypted network across all your devices. What that means is there’s no “relay” (like Wireguard server) and clients can talk to each other directly. While it may seem that it is easy to set this up with a bunch of shell scripting-foo (just add all Wireguard peers in each peer config), Tailscale attempts to do a lot of the heavy lifting by making the network seamless and handling authentication.

Coordination Server#

So, when we add a new device to Wireguard server, we basically need to generate a private/public key pair as explained above. When you’re setting up Tailscale agent, it does all of this in the background and asks you to authorise to a central co-ordination server. Now, this server is only used to exchange information about each peer in your network. The tailscale agent uploads the private/public key information of the peer you are currently on and any time any new peer joins the network, all of the agents’ configurations are updated real-time. The coordination server periodically checks for any new changes and pushes the updates to the agent on each node.

The real juice is that they authenticate via OAuth, OIDC or SAML. Which means that you can use your existing 2FA configurations to authenticate a new node in the network. However, this might be a point of concern for some users, but I chose convenience here. Also, since the traffic flows through the nodes directly, and is encrypted there’s not much to worry here. I’ve been following Tailscale closely and they do plan to opensource the co-ordination server in future, so maybe when that happens I’ll self-host it.

NAT Traversal#

Apart from handling the auth, Tailscale does a pretty good job at handling connectivity across different network topologies. So, basically, if you want to peer 2 devices over the public internet you’d need to perform NAT Traversal. There are several NAT hole punching techniques which allow traversing NAT but since they are not standardised and sometimes due to erratic NAT behaviour, it poses quite a challenge to do it seamlessly. And, I’m not even talking about roaming networks as of yet.

Tailscale agents can perform NAT traversal using ICE and STUN. What all of this practically means is if you’re sitting in a cafe somewhere and you want to access any of your internal service it is possible without messing around any firewall ports :D

TL;DR: I decided to give Tailscale a shot and was quite impressed by how easy it was to setup. You can refer to the official docs for more instructions but I was able to set it up across 2 RPis and have them talk to each other under 15 minutes. I think that’s quite an awesome feat in itself. The only sad bit is they don’t have an Android app yet, so I am eagerly waiting for it.

Hybrid K3s Cluster#

So, I am running a k3s cluster on my RPi. At that time I was still using DO for running Pihole, Unbound, public DNSCrypt resolver etc. I decided to standardise the ad-hoc deployments to manage them efficiently. It also allowed me to play around with more on K8s which was my original goal behind buying these Pi was.

Now, I’ve 2*RPi nodes, the k8s master node runs on the 4GB RPi while the 2GB variant serves as a worker node. I decided to get a bit fancy with my setup and hooked up the k3s agent installation script on a DO node andta-da! I have a multi-arch(amd64 and arm64), hybrid (bare metal and cloud) K8s cluster ready! I think if I sprinkle some ML/AI + Bitcoins in the above setup, I’m all set to raise VC funding for hydra.

I wanted to learn Terraform as part of my work at my org as well, so I created + managed the entire droplet through terraform. The script has modules to provision a droplet, attach an elastic IP, configure firewall rules and add my SSH key to the server. Quite easy to manage and I am generally a hater of GUIs, so Terraform is indeed a blessing in disguise.

I know some of my opinions are a bit strong but don’t worry I get meme’d/burnt for this almost every day by my work colleagues.

Pihole to Adguard#

While Pihole works really well for blocking Ads it had some features lacking, particularly DOT and DOH support out of the box. I decided I’ll shift to Adguard as the codebase is in Go - something which I am a bit familiar with and also the UI feels a bit sleek and refreshing too!

Accessing internal services#

A major challenge for me was however to configure access to internal services on the K8s cluster. Since I have bare metal worker nodes, it’s not possible to deploy a cloud load balancer. For now, I went with a really simple old school solution, to expose an Nginx proxy front-ending all my services through NodePort. I am planning to look at Traefik or Istio for this but I wanted to just shipit! at this point.

Here’s a very basic example of an nginx config for ip.mrkaran.dev that I run on my cluster:

server {
        server_name ip.mrkaran.dev;
        # reverse proxy
        location / {
                proxy_pass http://100.96.239.6:30506; # tailscale IP, connecting to NodePort service
                include fluff/proxy.conf;
        }
}

30506 port is exposed by the Service object backing up the pods for that application. Since a NodePort service is available on any K8s node, you can give the Tailscale IP for any node and the routing will be handled by kube-proxy.

Challenges faced#

Mesh or mess#

Setting all of this up didn’t come without its own challenges or hurdles. Right off the bat, the first problem was that I was seeing really high latencies from my RPi node to DO node through Tailscale. Now, since both the nodes are physically in Bangalore and they are connecting to each other (or that’s what I had presumed), I didn’t expect latencies to be as high as 500-600ms. Bollocks!

Eventually, I’d figured thanks to my super restrictive rules on the firewall of DO node, I had blocked all inbound UDP connection. That means NAT hole punching through STUN is simply not possible. In such cases, Tailscale forwards all packets to an encrypted TURN server(called DERP - Designated Encrypted Relay for Packets), which basically are TCP encrypted relays. Tailscale manages this network of DERPs and the one I got connected to was somewhere in the USA.

Bottom line, I was all pumped up to talk to a DO node from my RPi node but as it turns out (no pun intended) my packets were flowing through the USA! Ah so bad. Anyway, opening up UDP from the Tailscale subnet, fixed the issue and latencies were back to being sub 10ms. Yay!

Overlay over an overlay#

Next up, were problems with K3s networking from the DO node to RPi node. The DO node was in a NotReady state because the agent couldn’t reach the server:

Apr 14 09:00:33 hydra-control k3s[19746]: I0414 09:00:33.306650   19746 log.go:172] http: TLS handshake error from 100.97.222.106:51516: read tcp 100.96.239.6:6443->100.97.222.106:51516: read: connection reset by peer

Through some trial and error and reading the docs, I figured that flannel is running as a CNI in k3s. Now the problem is flannel itself is an overlay network. But… Tailscale already is an overlay network (Wireguard) so the packets are not being routed correctly and being dropped halfway in the master node (I am guessing the DNAT/SNAT translation botched up here).

The trick was to just change flannel backend to run in the host namespace only. That solved the above issue for me.

However, I still had one more issue. The DO node’s public IP was being advertised, while the agent was running on Tailscale network interface so the master was never able to reach the agent. Similarly, when the agent tried to communicate with the server, the private IP of the node was being advertised. Setting the --node-external-ip <tailscale-ip> in k3s config seemed to have fixed the problem.

Now all of the nodes in the cluster had proper Tailscale IPs advertised and the node went to Ready state, at last!

Who let the DNS out#

So, I’ve a chicken and egg problem in my setup. Since my laptop runs a Tailscale agent and whenever I boot up my systemm, Tailscale attempts to posts logs to log.tailsclae.io and fails to start if it cannot. The problem here is who resolves the DNS for me?

I run a local DNS server with CoreDNS forwarding my queries to Adguard. Now if I can’t reach Adguard (since Tailscale agent hasn’t initialised), how am I supposed to resolve log.tailscale.io? I did what any sane guy would do, write a simple hacky bash script:

#!/bin/bash
sudo chattr -i /etc/resolv.conf
sudo echo 'nameserver 1.1.1.1' > /etc/resolv.conf
echo "changed dns server to 1.1.1.1"
sudo tailscale up
sudo echo 'nameserver 127.0.0.1' > /etc/resolv.conf
echo "changed dns server back to 127.0.0.1"
sudo chattr +i /etc/resolv.conf

Yes, it’s quite insane. Also, I’ve not been able to figure out how to stop NetworkManager from changing my /etc/resolv.conf so I rely on a hack (documented in Arch official docs), to lock the file so any process cannot modify it. Quirky, but works!

Storage#

Unfortunately, I don’t have any external HDD/SSD so I am postponing running any stateful workloads till I get one soon (whenever lockdown gets over in my area). I plan to deploy an NFS server so I can run stateful workloads across any node and have redundancy in form of cloud backups. I’ve also heard cool things about Longhorn but unfortunately, it doesn’t have ARM support.

Final Thoughts#

Well, I am quite stoked by my current setup right now. Learnt a bunch of cool things around NAT traversal, zero-trust networking, realised the old days of LAN were so much better (not that I am that old to have experienced, my first internet connection was rather broadband at home, not even dial-up). Tailscale opens up a lot of new opportunities for corporate VPNs and it is something definitely to watch out for as they continue improving the product.

Also, I was super elated when bradfitz himself had commented on my tweet, not gonna lie!

{{< tweet 1249552822674149376 >}}

Credits#

First, a major thanks to my friend sarat with whom I pair debugged a lot of the above issues and since he also runs a home server, he was my go-to person to ask doubts!

Here are some links that I collected while setting all of this up and might be useful as references if you’re planning a similar setup!

• How Tailscale Works
• What are these 100.x.y.z addresses?
• k3s config reference
• P2P across NAT
• Remembering the LAN
• Cloudskew (wonderful tool for creating architecture diagrams)

If you’ve any questions please find me on Twitter @mrkaran. You can find all of the setup on GitHub.

Till next time, if we survive the pandemic!

]]> Karan Sharma https://www.prashanthudupa.com/upgrading-from-i-love-you-to-i-respect-you/ https://www.prashanthudupa.com/upgrading-from-i-love-you-to-i-respect-you/ Mon, 23 Mar 2020 09:40:43 GMT Prashanth Udupa Moments Unschooling https://www.prashanthudupa.com/webinar-on-unschooling/ https://www.prashanthudupa.com/webinar-on-unschooling/ Sat, 14 Mar 2020 13:16:06 GMT Prashanth Udupa Unschooling https://ibcomputing.com/change-font-in-telegram-desktop/ https://ibcomputing.com/change-font-in-telegram-desktop/ Tue, 10 Mar 2020 08:59:38 GMT Telegram Desktop is a wonderful application to access Telegram in computer. This applications is available for Gnulinux, Windows and Mac. But the main problem of Telegram Desktop is that, we cant change default font from the settings. this makes lot of readability issues in so many languages other than Latin based.  Let see how we can Change font in Telegram desktop application using some quick change in the system.

We have to change the font globally because Telegram doesn’t have the option to change font.

Change font in Telegram desktop application in Gnu-Linux OS

Create a file at the location ~/.config/fontconfig/fonts.conf

and add the following content to it. It sets default sans-serif font to Meera and serif font to Rachana. change the font name as your wish

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<!— —>
<fontconfig>
<!— Generic name aliasing —>
<alias>
<family>sans-serif</family>
<prefer>
<family>Meera</family>
</prefer>
</alias>
<alias>
<family>serif</family>
<prefer>
<family>Rachana</family>
</prefer>
</alias>
<alias>
<family>monospace</family>
<prefer>
<family>Liberation Mono</family>
</prefer>
</alias>
</fontconfig>

then run

 fc-cache -v.

and restart Telgram Desktop application. this will also change your fonts in all applications like browser.

In – Windows Operating System

Open Registy editor

press Win+R and type regedit

go to the location given below

HKEY_LOCAL_MACHINE
SOFTWARE
Microsoft
Windows NT
CurrentVersion
FontSubstitutes

Change Value of “MS Shell Dlg 2” to the font name

I don’t have a Mac system, if anyone of you know about it add it in comments. This will change the fonts in Telegram Desktop application.

 

The post How to change font in telegram desktop in Linux and Windows appeared first on IB Computing.

]]> Mujeeb cpy Telegram Tutorials GNU/Linux Windows https://ibcomputing.com/build-presentations-with-reveal-js-and-markdown/ https://ibcomputing.com/build-presentations-with-reveal-js-and-markdown/ Sat, 22 Feb 2020 06:13:44 GMT Presentations are important to share something to others. we have Office software such as Libre office impress, Microsoft Office PowerPoint, google Slides to do this task. Today I am going to tell you how you can Build Presentations with Reveal.js and markdown file.

Reveal js is a JavaScript package to create web slides.  official website itself a presentation demo.  it also provides a free online editor to create slides . but it needs an account to start. just go to slide.com for making one.  lets see how we can create a web slide using a markdown file.

Build Presentations with Reveal.js and markdown

Before we start we have to install the following packages into the system.

execute these commands to install the packages in Ubuntu. Other operating systems have there own package manager commands.

git : sudo apt-get install git
pandoc : sudo apt-get install pandoc

step 1 : Create a Markdown File with headings and list items. an example here

django-intro.md

% Django Introduction
% Mujeeb Rahman K
% February 09, 2020

# Django

## Why Django
* Django is a Web framework written in Python
* Don't reinvent the wheel.
* 216 K packages in python package index

step 2 : clone or download Revealjs into the same folder which contains the above markdown file

git clone https://github.com/hakimel/reveal.js.git

Step 3 : convert markdown (.md) file to html file using pandoc utility

pandoc -t revealjs -s -o djangoslide.html django-intro.md -V revealjs-url=./reveal.js

This will convert md file to html and use reveal js from our current folder. if you don’t cloned reveal js you can use direct url like below.

pandoc -t revealjs -s -o djangoslide.html django-intro.md -V revealjs-url=https://revealjs.com

now we have got a djangoslide.html file. just open that file in any browser and see the magic..!! your web slide is ready.  remember that, if you didn’t download or clone reveal js in same folder the html wont render correctly.

The is a very quick and simple way to create a presentation because it only need very simple utilities like git, pandoc and a simple markdown file. Markdown can be created using a very simple text editor. We have very nice presentation with all the effects. It is html so simply works everywhere.

The post Create A Simple Presentation using Reveal js and Markdown appeared first on IB Computing.

]]> Mujeeb cpy Tutorials Javascript Markdown Presentation revealjs https://mrkaran.dev/posts/ndots-kubernetes/ https://mrkaran.dev/posts/ndots-kubernetes/ Sun, 02 Feb 2020 05:27:55 GMT One of the primary advantages of deploying workloads in Kubernetes is seamless application discovery. Intra cluster communication becomes easy with the concept of Service which represents a Virtual IP backing a set of Pod IPs. For example, if vanilla service needs to talk to a chocolate service, it can directly use the Virtual IP for chocolate. Now the question is who resolves the DNS query for chocolate and how?

DNS resolution is configured in Kubernetes cluster through CoreDNS. The kubelet configures each Pod’s /etc/resolv.conf to use the coredns pod as the nameserver. You can see the contents of /etc/resolv.conf inside any pod, they’ll look something like:

search hello.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.152.183.10
options ndots:5

This config is used by DNS clients to forward the DNS queries to a DNS server. resolv.conf is the resolver configuration file which has information about:

• nameserver : Where the DNS queries are forwarded to. In our case this is the address of CoreDNS service.

• search: Represents the search path for a particular domain. Interestingly google.com or mrkaran.dev is not FQDN (fully qualified domain name). A standard convention that most DNS resolvers follow is that if a domain ends with . (representing the root zone), the domain is considered to be FQDN. Some resolvers try to act smart and append the . themselves. So mrkaran.dev. is an FQDN but mrkaran.dev is not.

• ndots: This is the most interesting value and is the highlight of this post. ndots represents the threshold value of the number of dots in a query name to consider it as a “fully qualified” domain name. More on this later, as we discover the flow of DNS lookup.

Let’s check what happens when we query for mrkaran.dev in a pod.

$ nslookup mrkaran.dev
Server: 10.152.183.10
Address: 10.152.183.10#53

Non-authoritative answer:
Name: mrkaran.dev
Address: 157.230.35.153
Name: mrkaran.dev
Address: 2400:6180:0:d1::519:6001

For this experiment, I’ve also turned on CoreDNS logging level to all which makes it highly verbose. Let’s look at the logs of coredns pod:

[INFO] 10.1.28.1:35998 - 11131 "A IN mrkaran.dev.hello.svc.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.000263728s
[INFO] 10.1.28.1:34040 - 36853 "A IN mrkaran.dev.svc.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000214201s
[INFO] 10.1.28.1:33468 - 29482 "A IN mrkaran.dev.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000156107s
[INFO] 10.1.28.1:58471 - 45814 "A IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 56 0.110263459s
[INFO] 10.1.28.1:54800 - 2463 "AAAA IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 68 0.145091744s

Whew. So 2 things piqued my interest here:

• The query iterates through all search paths until the answer contains a NOERROR code (which the DNS clients understand and store it as the result). NXDOMAIN simply indicates no record found for that domain name. Since mrkaran.dev isn’t an FQDN (according to ndots=5 setting), the resolver looks at search path and determines the order of query.

• A and AAAA records are fired parallelly. This is because single-request option in /etc/resolv.conf has a default configuration to perform parallel IPv4 and IPv6 lookups. You can disable this using single-request option.

Note: glibc can be configured to send these requests sequentially but musl cannot, so Alpine users must take note.

Playing around with ndots#

Let’s play around with ndots a bit more and see how it behaves. The idea is simple, for the DNS client to know whether a domain is an absolute one or not is through this ndots setting. For example, if you query for google simply, how will the DNS client know if this is an absolute domain. If you set ndots as 1, the DNS client will say “oh, google doesn’t have even one 1 dot, let me try going through search list. However, if you query for google.com, the search list will be completely ignored since the query name satisfies the ndots threshold (At least one dot).

We can see this by actually doing it:

$ cat /etc/resolv.conf
options ndots:1

$ nslookup mrkaran
Server: 10.152.183.10
Address: 10.152.183.10#53

** server can't find mrkaran: NXDOMAIN

CoreDNS logs:

[INFO] 10.1.28.1:52495 - 2606 "A IN mrkaran.hello.svc.cluster.local. udp 49 false 512" NXDOMAIN qr,aa,rd 142 0.000524939s
[INFO] 10.1.28.1:59287 - 57522 "A IN mrkaran.svc.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000368277s
[INFO] 10.1.28.1:53086 - 4863 "A IN mrkaran.cluster.local. udp 39 false 512" NXDOMAIN qr,aa,rd 132 0.000355344s
[INFO] 10.1.28.1:56863 - 41678 "A IN mrkaran. udp 25 false 512" NXDOMAIN qr,rd,ra 100 0.034629206s

Since mrkaran didn’t specify any . so the search list was used to find the answer.

Note: ndots value is silently capped to 15 and is 5 in Kubernetes as default.

Handling this in Production#

If your application is of the nature that makes a lot of external network calls, the DNS can become a bottleneck in case of heavy traffic since a lot of extra queries are made before the real DNS query is even fired. It’s quite uncommon to see applications appending the root zone in the domain names, but that can be considered as a hack. So instead of using api.twitter.com, you can hardcode your application to include api.twitter.com. which would force the DNS clients to do an authoritative lookup directly on the absolute domain.

Alternatively, since K8s 1.14, the dnsConfig and dnsPolicy feature gates have become stable. So while deploying a pod you can specify ndots setting to something lesser, say 3 or if you want to be really aggressive you can turn it down to 1. The consequences of this will be that every intra-node communication now has to include the full domain. This is one of the classic tradeoffs where you have to choose between performance and portability. If the app doesn’t demand super low latencies, I guess you need not worry about this at all since DNS results are cached internally too.

References#

I got to know about this peculiarity first, in a K8s meetup which I went to, last weekend where the folks mentioned about having to deal with this.

Here are some additional links you can read on the web:

• Explainer on why ndots=5 in kubernetes
• Great read on how ndots affects application performance
• musl and glibc resolver inconsistencies

Note: I’m particularly not using dig in this post. dig apparently automatically adds a . (root zone identifier) to make the domain an FQDN one without even first going through the search path. I’ve mentioned about this briefly in one of my older posts. Nonetheless, it’s quite surprising to see that you need to give a flag to make it behave in what seems to be a standard way.

It’s always DNS, isn’t it ;)#

Fin!

Update#

2020-12-24:

In case you want to play around with ndots and DNS more, I’ve worked on a DNS Client which lets you tweak these params on the fly. Feel free to checkout doggo if interested!

]]> Karan Sharma https://kcsrk.info/ocaml/prolog/jupyter/notebooks/2020/01/19/OCaml-Prolog-Jupyter/ https://kcsrk.info/ocaml/prolog/jupyter/notebooks/2020/01/19/OCaml-Prolog-Jupyter/ Sun, 19 Jan 2020 15:16:00 GMT Last semester at IIT Madras, I taught a revamped core course CS3100 Paradigms of Programming, which introduces 3rd-year students to functional and logic programming paradigms. While the course had been traditionally offered in Lisp and Prolog, I introduced OCaml instead of Lisp. All of the lectures were delivered through interactive Jupyter notebooks. The assignments were also distributed as Jupyter notebooks and evaluated through autograder facility in Jupyter. There has since been several requests to replicate this setup elsewhere. Hence, I thought I should write about the set up and experience of teaching through Jupyter notebooks.

Course Content

Having never taken a functional programming course, there was the question of what I wanted the students to take away from the course. I wanted the course to be a mixture of functional programming concepts (types and lambda calculus) as well as advanced yet pragmatic concepts that one would find in modern functional programming languages (such as GADTs and Monads). The OCaml part of the course is based on the excellent CS3110 from Cornell and AFP from Cambridge Computer Laboratory. In particular, I would highly recommend the CS3110 book for anyone taking first steps into functional programming. Lambda calculus lectures were based on Peter Selinger’s lecture notes on lambda calculus.

The Prolog part of the course were modelled on Prolog lectures from Cambridge Computer Laboratory and the wonderful The Art of Prolog book.

Teaching functional and logic programming in the same course allowed me to develop interesting content that intersected both of the paradigms. In the functional programming part of the lecture, I had introduced simply typed lambda calculus. In the logic part of the course, we developed a type checker for simply typed lambda calculus in Prolog. Merely encoding type checking rules for simply typed lambda calculus in Prolog, type inference with polymorphic types falls out. With a tiny bit of coaxing, Prolog synthesizes programs for the given type. In the last assignment, the students were asked to implement a Prolog interpreter in OCaml. There was indeed some value in teaching multiple paradigms in the same course, not just for a comparative study of strengths and weaknesses, but to be able to teach the students to pick the right tool for the job.

Course Delivery

I had a clear idea that the course will have to be interactive where programs are developed during the lectures. There was the option of using pdf slides and switching to utop for interactive development. But this solution lacked the uniformity that the students would like when reviewing the course materials. Moreover, switching between two mediums made it difficult for me to plan the lectures and was a distraction for the students.

Jupyter Notebooks

Hence, I decided to use Jupyter Notebooks for the course. Jupyter is a collection of open source standards and software for interactive development. Jupyter supports a variety of languages. For OCaml, I used akabe/ocaml-jupyter, an OCaml kernel for Jupyter notebooks. This uses utop, an advanced OCaml top-level in the backend and hence provides excellent interactive top-level support. The situation for Prolog was not so great. Eventually, I zeroed in on targodan/jupyter-swi-prolog but ended up improving the solution a bit kayceesrk/jupyter-swi-prolog (TODO KC: upstream fixes). Jupyter supports mathjax, which allows typesetting LaTeX in the notebooks. This was great for writing the lectures on lambda calculus.

RISE for slideshow

Jupyter notebooks are webpages that mixes text and code. For lectures, I much prefer slides since they let you focus on a particular images, statement or an inference rule. While Jupyter allows the conversion of notebooks to slides out-of-band, RISE is an Jupyter notebook extension that lets turn your Jupyter notebook into a slideshow. Adding RISE to the setup makes the Jupyter experience compatible with traditional slides based lectures.

Course Distribution

Apart from delivering the lectures through the notebooks, I also wanted the students to be able to go through the notebooks and be able to run the snippets. Installing all the required software (OPAM, OCaml, Prolog, Jupyter and its extensions, Jupyter Kernels for OCaml and Prolog) and correctly was not something I wanted the students to go through. I wasn’t even sure if this software combination works on various Mac, Windows and Linux distributions. Hence, everything was packaged as a Docker file, and the latest version of the image uploaded to docker hub. In order to review the course, the students only had to install Docker and Git and run exactly 4 commands.

Docker is generally supported on all major OSes. Packaging up the course content as a docker image and pushing it to dockerhub is insurance against the software combination not working in the next offering of the course; if for some reason one of the dependency does not work next year, I can always fallback to the docker image while I find a fix. One of my TAs ran a tutorial on basic Docker and Git in the first week of the course to ensure that everyone was setup. I would consider Docker and Git as essential tools for modern software development as well as research. After that, the students did not ever have to do anything on the command line.

Assignments

nbgrader is a tool that facilitates creating and grading assignments in Jupyter notebook. It uses language-agnostic logic to identify failing cells, which meant that it was easy to set up nbgrader for OCaml and Prolog. The assignments were released as Jupyter notebooks, which the students filled in and submitted. nbgrader has support for unit tests which allowed the students to get instant feedback as they were developing the solutions.

Wish List

Overall, the students felt that the Jupyter notebooks were better than slidedecks. However, not everything was perfect with the Jupyter notebook based lecturing. Here are some of the things that could be improved.

• There is no good diagramming + animation support for Jupyter notebooks. The best I could find was egal whose user interface I did not find intuitive. Even for simple diagrams, it was much more effort making diagrams there compared to Keynote, PowerPoint or OmniGraffle. Eventually, I used draw.io to make the diagrams and include the images in the slides for a few of the cases where I actually needed to make diagrams.
• Docker for Windows does not work on Windows Home or Student. Support for OPAM on Windows is slowly improving, but it is not yet for novices. Hence, I had to recommend the students to run an Ubuntu VM on their Windows machines in which they ran the course’s docker container.
• nbgrader had several bugs which caused the autograder to award marks even for failing cells. The TAs had to go through a few of the assignments manually to ensure that students were awarded grades correctly. This is something that should be fixable easily.
• RISE doesn’t easily let you change the size of the font. One has to edit the CSS to change the font size. And the default style wastes too much space. This meant that not much content can be fit into a single slide. Hence, I’ve had to artificially split content into multiple slides or zoom out several steps to show content that was cut off on the bottom.
• The support for Prolog is not so great. There are a few advanced features in Prolog for which the Prolog setup fails. I had to switch to SWI-Prolog top-level for a few lectures. That said, the Prolog support is mostly there and the issues can be fixed with some effort.

Conclusion

I have started working on fixing some of these issues and upstreaming the solutions. Hopefully the fixes should be ready for the next iteration of the course. If you would like to replicate this setup for your course, do feel free to utilise the course materials.

]]> KC Sivaramakrishnan https://mrkaran.dev/posts/resize-pvc-k8s/ https://mrkaran.dev/posts/resize-pvc-k8s/ Tue, 14 Jan 2020 02:40:55 GMT Well, the title is self explanatory so let’s begin!

First off, we need to ensure that the StorageClass which was used to provision the PVC has the correct configuration. From the official docs:

You can only expand a PVC if its storage class’s allowVolumeExpansion field is set to true.

So, let’s inspect our storage class:

$ kubectl get sc # sc is short for storageclass
NAME            PROVISIONER             AGE
gp2 (default)   kubernetes.io/aws-ebs   8d

$ kubectl describe sc/gp2
# output redacted to focus only on the field we're concerned with
Name:            gp2
AllowVolumeExpansion:  True

If AllowVolumeExpansion is set to True you can skip the below step. If it’s not true, you need edit the field allowVolumeExpansion as true:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: gp2
parameters:
  fsType: ext4
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Once the StorageClass config is correct, all we need to do is update with the new size.

So, for example if size of the PVC was 15GB orginally:

spec:
  resources:
    requests:
      storage: 15Gi

To update it to 30GB, you simply need to edit spec.resources.requests field:

spec:
  resources:
    requests:
      storage: 30Gi

We now need to “apply” the updated PVC manifest.

$ kubectl apply -f pvc.yml

Let’s take a look at the PVC:

$ kubectl describe pvc/myclaim
# output redacted for brevity
...
Conditions:
  Type                      Status  LastProbeTime                     LastTransitionTime                Reason  Message
  ----                      ------  -----------------                 ------------------                ------  -------
  FileSystemResizePending   True    Mon, 01 Jan 0001 00:00:00 +0000   Tue, 14 Jan 2020 20:52:21 +0530           Waiting for user to (re-)start a pod to finish file system resize of volume on node.
...

So, basically what FileSystemResizePending means is that while PVC is in use, we have to either restart or delete the underlying Pod using the PVC. At the time of writing this, ExpandInUsePersistentVolumes is still in beta and has to be enabled as a feature gate. Sadly, EKS is still on 1.14 (while the world has moved to 1.17, such sloooow release cycles!), so I couldn’t enable this in my case.

Once the pod is restarted, the expanded disk is automagically available! Let’s verify this:

kc get pvc/myclaim -o=jsonpath="{.status.capacity.storage}"
30Gi

Now, compare this with the standard way of resizing an EBS volume in EC2 instance. You need to first modify the volume size using AWS EBS API and then in the EC2 instance, use a combination of growpart and resize2fs to extend the resized volume. This sounds much more cumbersome than simply updating the storage field in PVC manifest!

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/using-sshuttle/ https://mrkaran.dev/posts/using-sshuttle/ Sun, 12 Jan 2020 04:27:55 GMT :{upstream-host}:{upstream-port} user@remote The problem with ssh tunnels is that they experience frequent packet loss on a normal WiFi connection and it’s quite frustrating to deal with them. Moreover, sometimes you need access to multiple ports in your private network which requires you to explictly provide them with -L flag which I find it as cumbersome. Also, you cannot forward DNS queries (over UDP) since ssh can only do TCP. sshuttle has made my life so simple! Fin!]]> The Motivation#

Sometime back I had to access a Kubernetes API server which was firewalled to a private VPC network. I didn’t want to setup a separate bastion instance just to access this cluster, cause TBH bastions are kinda redundant in K8s as every task can be performed through the client-server APIs using kubectl. So, all I needed was access to this API server from a trusted network in a secure way. Thanks to my friend @sarat, I got to know about sshuttle. sshuttle is quite unique in the sense that it’s not really a VPN but acts like one (for most practical purposes). sshuttle lets you access an internal network through a trusted node inside the VPC, without you having to deal with the mess of port forwarding or VPNs.

The basic idea is pretty simple, sshuttle starts a local python server in your host machine and creates iptables rules to route the destination packets of the specified CIDR blocks to this local server. At the server, the packets are multiplexed over an ssh session and sent to the server. The server disassembles the multiplexed packet and the routes them to upstream. So, basically this is a clever hack to avoid TCP over TCP (which again is a mess on unreliable networks). Multiplexed streams on ssh is just a single stateful TCP connection (as compared to VPN connections which are stateless). Now you must be wondering, how come the target server disassembles the packets. Yes, there needs to be some kind of sshuttle daemon running which does that for you. This is where sshuttle does some magic, it automagically deploys a python script on your target host to perform this task. So yes, for sshuttle to work, both the client and target need to have python and iptables installed.

Usage#

sshuttle -r user@port x.x.x.x

All the packets routed to the CIDR block will now go through sshuttle daemon, since it configured iptables rules for them. Also, sshuttle starts a local python server on your host machine. You can see it using netstat:

$ sudo netstat -tunapl | grep python
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:12300         0.0.0.0:*               LISTEN      27425/python

There’s a python server listening on port 12300 in my host machine. To actually verify, this indeed is started by sshuttle, you can use pstree -p | less and search for sshuttle. Here you can see sshuttle did indeed start a python server and the PID (27425) matches with the one we saw in netstat command.

    -zsh(13201)---sshuttle(27425)-+-ssh(27446)---ssh(27447)
                                    `-sudo(27427)---python(27445)

You can even forward DNS queries with the --dns flag. This is super helpful if you have something like Route53 to host your DNS records on a private zone (for eg tld like .internal).

Better than SSH tunnels?#

Yes, you can also port forward with ssh using: ssh -nNT -L <local-port>:{upstream-host}:{upstream-port} user@remote

The problem with ssh tunnels is that they experience frequent packet loss on a normal WiFi connection and it’s quite frustrating to deal with them. Moreover, sometimes you need access to multiple ports in your private network which requires you to explictly provide them with -L flag which I find it as cumbersome. Also, you cannot forward DNS queries (over UDP) since ssh can only do TCP.

sshuttle has made my life so simple!

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/netcat-port/ https://mrkaran.dev/posts/netcat-port/ Sat, 11 Jan 2020 02:40:55 GMT Quite often you’d need to check if a port on a target node is opened or blocked by firewall. I’ve always used telnet to test that but it has a few drawbacks:

• Need to use dirty hacks in shell scripts to auto close the connection. Also, telnet outputs some errors to /dev/stdout instead of the standard /dev/stderr which makes it harder to use in scripts.

• Non standard implementation across different OSes. On Alpine Linux (mostly used in containers), if you install telnet using the /busybox-extras package, the behaviour is different from what it is on standard Ubuntu/Arch environments. I’ve even faced weird issues on Alpine where telnet will simply wait endlessley for the connection to be established, while netcat would not indicate any issues.

• Telnet is actually a protocol and the telnet-client initiates the negotiation with the server before a connection is established.

So, after all these issues, I looked at other tools to eventually replace telnet with something better. I tried nmap which is also a port scanner, but is unreliable since a lot of hipster sysadmins drinking the security koolaid block port scanning tools like these. I wanted a dependable tooling and after a bit of Google-fu, I stumbled across netcat.

netcat is basically a swiss army knife to perform all kind of ops with TCP/UDP. You can create a file server, chat client/server, TCP client etc. We are simply interested in the port scanning abilities of this for this blog post, so let’s actually see how to use it for the same.

Note: Install netcat-openbsd as it is a rewritten version of netcat-traditional with some more bells and whistles.

The basic syntax for port scanning looks like:

nc -z host port

-z tells nc to not send any data, just scan for any process listening on the target port. This is much better (and faster) than telnet client initiating a connection with the upstream.

To make it more usable however, let’s pepper our command with some helpful flags:

nc -vz -w 3 host port

-v turns on verbose mode which outputs diagnostic messages. -w adds the timeout for the connection to be established. If you want to set a timeout in telnet there’s a hack for it.

You can even supply a range of ports to netcat like:

nc -vz -w 3 host 8000-9000

Quick Tip: You can also give an alias for port instead of the number. For example:

$nc -vz -w 3 google.com https
Connection to google.com 443 port [tcp/https] succeeded!

$nc -vz -w 3 google.com ssh
nc: connect to google.com port 22 (tcp) timed out: Operation now in progress
nc: connect to google.com port 22 (tcp) failed: Network is unreachable

Hope this post pretty much sums up the usage of netcat for port scanning! Read the man page for more info.

Fin!

]]> Karan Sharma https://captnemo.in/blog/2020/01/04/security-setup/ https://captnemo.in/blog/2020/01/04/security-setup/ Sat, 04 Jan 2020 00:00:00 GMT I upgraded my encryption setup recently, so I thought I should write about it, just in case it is helpful to someone else. As a security professional, I have a different threat model from most folks, and as such my setup does involve a bit more complexity than what I’d recommend to everyone. But if you are an at-risk individual (journalist, person holding hundreds of bitcoins or other digital assets, activist) or if you are a linux user with a lot of free time - you might consider duplicating some of this.

I’ll discuss some of the other approaches I’ve considered, and my thought process around each choice I made. There are general recommendations at the bottom of the post.

Passwords

I used to be on LastPass till 2017¹, when I migrated to [pass][pass] (“the standard unix password manager”). With pass, each password lives inside of a gpg encrypted file whose filename is the title of the website or resource that requires the password.

pass automatically manages a git repository for you, which I sync against GitLab. The one downside of using pass is that the list of my domains is visible to my hosting provider. ² In the past, I’ve set this up against Keybase Encrypted Git (Keybase doesn’t get to see even the file list), and my own git-server (only I get to see it).

I don’t push it to GitHub, since most of my stuff lives on GitHub anyway, and I didn’t want to add my passwords there as well. GitLab uptime is decent enough for my usecase³. Finally, my GitLab account is fairly locked down with:

• zero integrations or third-party apps
• no active personal tokens
• social signin disabled
• only yubikey SSH key configured

Mobile Passwords

There are 2 primary considerations I have:

1. Using pass involves GPG keys, and I can’t use hardware GPG keys on my current device (iPhone SE).
2. I don’t want to sync all my passwords to my phone. I have a limited number of applications on my device, and syncing all passwords doesn’t make sense.

For the first issue, I am forced to use a PGP key in software on [passforios][passforios]. If you are on Android, take a look at [OpenKeychain][okc], and Fidesmo/Yubikey NFC.

For the second issue, I use pass cp, and the .gpg-id file, which allows me to maintain a mobile-sync directory inside my pass git repository encrypted against a different key. From the pass documentation:

~/.password-store/.gpg-id

Contains the default gpg key identification used for encryption and decryption. Multiple gpg keys may be specified in this file, one per line. If this file exists in any sub directories, passwords inside those sub directories are encrypted using those keys. This should be set using the init command.

My ~/.password-store/mobile-sync/.gpg-id file holds 2 keys: My main encryption key, and the key I’ve configured on my phone.

Unfortunately, I haven’t gotten it working well as a git submodule, so I have a helper script that copies the encrypted password files from mobile-sync subdirectory to a different repository (mobile-passwords.git). The script is just 2 lines:

cp -r ~/.password-store/mobile-sync/*.gpg .
git-sync

It updates the git repository, and runs a sync to push any local changes to my mobile-passwords repository. I can pull that on my passforios application. I could also clone the entire repo, but the iOS app doesn’t work nicely with a single-subdirectory approach.

GPG

pass relies on GPG, and as such I require a strong key setup. I have the following:

1. 2xYubikey 4 (Doesn’t have NFC)
2. Fidesmo Smartcard, currently unused

Both the Yubikeys are configured against my GPG Encryption key. I carry one of the Yubikeys on my keyring with me. The backup Yubikey stays at my home.

I followed this guide while configuring the same. As of now, switching between keys is not very user-friendly, but future GnuPG versions plan to fix it. The Yubikey holds:

• An encryption key
• A signing key
• An authentication key

I keep a copy of all these keys using paperkey as per the same guide. I have a subkey backup as well, since Yubikeys are known to fail⁴.

SSH

The Authentication key in my Yubikey is configured for SSH. I just need to ensure that my GPG agent is configured for SSH as well:

export GPG_TTY="$(tty)"
export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
gpgconf --launch gpg-agent

U2F

U2F lets me use a physical key as my second-factor on supported websites (as an alternative to SMS/TOTP). I configure both of my Yubikeys for U2F wherever possible (Twitter/AWS are notable exceptions and only support a single key). U2F support for [OpenSSH][ssh-u2f] is coming soon. So you can soon authenticate to your server with the Yubikey+PIN, and finish the 2FA with U2F by tapping the key⁵.

Root-of-Identity

For most people, the root of identity comes down to ownership of their email address. As such, it is very often the juiciest target for most attackers. I run my mail against Migadu, a privacy friendly swiss email-hosting service. They provide me a management layer for managing my domains which uses my GMail account. (See FAQ for why). I also have 2FA (TOTP only) configured on the Migadu management setup.

The domain is currently registered at a Indian registrar (which doesn’t offer U2F, but I do have TOTP configured). I would have moved this to CloudFlare, but CloudFlare doesn’t support the .in TLD yet.⁶

The email address used for registring the domain again is my GMail. My DNS is configured on CloudFlare, which again uses my GMail and has appropriate 2FA configured. (It doesn’t support U2F). So here’s the list of critical providers:

Provider	What can an attacker do	Auth	2FA
Regisrar/Mitsu	Change my nameserver, read any future email, reset passwords	GMail/Password	TOTP
DNS/CloudFlare	Change my MX records, read any future emails, reset passwords	GMail/Password	TOTP
Email/Migadu	Reset my email password, read my current emails, reset passwords	GMail/Password	TOTP
GMail	Reset passwords for the above 3 accounts	Email/Password	U2F

As you can see, I end up trusting GMail a lot here.

Recovery/Backup codes and Security Questions

I use randomly generated UUIDs as answers to security questions. Currently, I’m storing these for various services within the same password store. As it stands, I can’t get access to my password OR recovery token without my Yubikey and PIN. The access Matrix looks like this:

What	Physical Access	Additional Authentication
Password	Yubikey	PIN
U2F-2FA	Yubikey	Physical touch
TOTP	Phone	TouchID
Recovery Code	Yubikey	PIN
Security Question answers	Yubikey	PIN

Failure Scenarios

There are lots of failure scenarios with such a setup, and while I’ve got a pretty spotless record of not getting hacked - I’m not immune to screwups. Here’s all the bad things that can happen:

Yubikey failures

If my Yubikey fails (or if I forget its PIN), I can’t access passwords on my device. I still have access to commonly used passwords and my mail on my phone. My backup Yubikey is kept safely at my home. If I lose both, I have the paperkey backup at home (which I should store elsewhere).

Device failures

I have a current version of the password repository against 3 PCs, and a partial version in my mobile. If all these 4 devices fail at once, I can still clone a fresh version of the repository with my YubiKey (I would still have GitLab SSH access). I might need to prepare better for this though, since configuring GPG-SSH might not always be easy during an incident.

As a alternate scenario, my phone GPG key does have my GitLab password, so I can clone the repo over HTTPS (with password) if needed.

Circular dependency against GitLab

My GitLab password is randomly generated and stored in the same password store on GitLab. That is not too big of an issue, because I don’t need the GitLab password anymore to clone my passwords repo, just my Yubikey (for SSH).

Lost Key

If I lose my key, the GPG card contains public info, including my email address, which can be used to contact me. I have a [Tile bluetooth tracker][tile] on my keychain to make it easier for me to find it.

Malware

A hardware key doesn’t protect you from all attacks. At the end of the day, my passwords must be decrypted by the key and passed unencrypted back to my browser (or editor). pass for eg, doesn’t protect against memory scraping attacks. If I edit a password on an infected machine, it gets that password.

If my browser has a malicious extension, it already has keys to the kingdom. But if I then log into a website, it does get access to that password additionaly.

xkcd 1200 famously illustrates this:

A password vault protected by a hardware key protects against some attacks:

• A malicious extension can’t sniff my vault passphrase, since I don’t have one
• The key can’t be exfiltrated from hardware.

However, a malware can connect to my authenticated GPG socket, and start decrypting things. To prevent against that, I run my Yubikey in “touch-only” mode, so it requires a “physical touch” before it actually does anything, even if the PIN is cached. Customizability is dependent on your Yubikey model. But remember the xkcd warning - if I have a malware running on my device, it is pretty much game over anyway. pass doesn’t prevent against memory scraping attacks, and actually uses /dev/shm to store the temporary plain-text files containing passwords. Ultimately, your identity is as secure as the device you trust it with.

Improvements

If you have any suggestions for any of the below, I’m happy to hear them.

Travel Plans

Since my backup key stays at home, how do I deal with long-term travel? This is something I’m still figuring out. Do I take my backup Yubikey on my longer-travels? Or should I setup a third-key before I do that? Chances of me losing both the keys together are quite high, so I’m trying to avoid that.

Domain Ownership

I’d like to transfer my domain to a registrar that supports U2F, likely Namecheap since I already own some domains there⁷. If you use CloudFlare, they should roll out U2F support soon.

2FA Recovery Guides

I wish more organizations published what they consider as valid 2FA recovery mechanisms. GitHub supports 2FA recovery by proof of SSH keys or Personal Tokens; Migadu just needs a few domain names from your account, and lots of services require proof-of-identity.

A lot of this is undocumented, and I wish organizations were more public about this so users can take appropriate measures and understand their risk better.

Fidesmo Card

I’m planning to configure my Fidesmo card against my existing GPG/SSH key, so it stays in my wallet to improve redundancy. Unfortunately, it is not supported on iOS, so I plan to get a NFC reader/writer and test that out. This also helps with travel plans a bit, since I’m less likely to lose my wallet anecdotally(which also has [a bluetooth tracker][slim]).

U2F on iPhone

U2F support on Mobile Safari is non-existent. Brave recently added support for the upcoming Yubikey 5Ci, which supports both USB-C and lightning. However, this requires a special Yubikey SDK, which breaks the idea of U2F being interoperable. The 5Ci is also quite costly at $70. I don’t know of any application that is actually supporting GPG-over-Yubikey-over-lightning.

Compare this to Android where NFC based smartcards or Yubikeys just work. I’d like that to happen with iPhones.

Full Disk Encryption using a Yubikey

It is possible to configure Full Disk Encryption with Yubikeys, but I haven’t tried it yet.

2FA on my email account

Migadu currently does not support 2FA on webmail access, just on manage.migadu.com. This is very unfortunate, but I’m told this is planned soon (January 2020).

Recovery/Backup codes and Security Questions

My current setup of saving recovery codes alongside passwords isn’t optimal, but I don’t have a better way either. I’ve considered keeping my recovery codes on a alternate password store (such as bitwarden, or keypassX), but I’ll have to memorize the password, and setup a separate 2FA for it to be truly fault-tolerant.

FAQ

Why do you have stuff configured on your GMail? Aren’t you anti-Google?

Despite all the flak that Google gets for privacy, their security team is pretty awesome. Your account is pretty much unhackable once you are enrolled into their Advanced Protection Program. The few security-sensitive places where I use it are:

1. Domain Registration
2. Email Management
3. DNS Configuration

Everywhere else, I use my actual domain (captnemo.in) to ensure nothing else routes over GMail. Using GMail for the above 3 ensures that I don’t have a circular dependency. If I were to lose my main email password, I can recover via multiple ways:

1. Change DNS to another email provider.
2. Reset password via migadu admin panel.

Ensuring that either of these workflows do not rely on the same email account I’ve just lost access to is vital. Another alternative is to use a trusted-friend (ideally someone more paranoid than me) as a proxy for these emails, and use their domain for managing these 2 services. Might get around to it someday.

My GMail recovery email is set to my main account, so it creates a circular dependency, but one that I actually want.

What do you recommend I use?

• Bitwarden for password management.
• 2xHyperFIDO Mini U2F Keys configured for second factor against as many accounts as possible. U2F is not only safer, but much more convenient than TOTP/SMS based 2FA. For iPhone/USB-C users, see the Yubico website. If you don’t like to pay the USB-C tax, there are cheap USB-C to miniUSB adapters that can work with the HyperFIDO key and fit on your keychain. If you aren’t convinced on why this is a good idea, see this guide. The second key is just a backup key, and could be the primary key used by your spouse, friends or co-workers.
• A PIN configured on all your SIMs. Instructions for iPhone, Android.
• Full-Disk-Encryption on all your devices. Instructions for Windows, Mac, ArchLinux, Fedora, Ubuntu.
• Use randomly generated passwords everywhere. Trust your password manager on this.
• Setup a PIN on your WhatsApp.
• If you own a lot of cryptocurrency, use a hardware wallet and put it in a bank safe. Have a backup one, in another safe. You can put the PIN for those in your password store. I haven’t researched enough to suggest you which wallet(s).
• Get a SIM without an Aadhaar, to make SIM-Jacking attackes harder (applies in India).
• Go through securityplanner.org, which gives you personalized recommendations customized for our risk profile. I agree with most of their recommendations⁸
• Signup for breach notifications against your email at https://haveibeenpwned.com/.
• If you’d like to get off GMail, pay for FastMail. Alternatively, if I know you in real-life, I’m happy to host your mail in my Migadu account. (Only works if you know me well enough to trust me)

Why are you so paranoid?

I work in infosec. Breaking things comes naturally to me, and I plan for defense-in-depth. Plus, I’d be a terrible security person if I got hacked.

Why not recommend open source keys instead?

Availability is a pain point, especially if you aren’t in the US. Even getting my hands on a SoloKey was hard, despite backing it on KickStarter.

• OnlyKey also makes some claims regarding open source, but I can’t find their schematics anywhere.
• SoloKey is great, and what I’d recommend, but it doesn’t support OpenPGP yet.
• [NitroKey Start][nitrokey-start] is apparently completely FOSS, so you might wanna check that.

The HyperFIDO keys are compliant to the U2F/FIDO standards, and I’ve not faced any issues while using them. They’re cheap and widely available. Unless you need GPG, go for it.

---

Thanks to Giridharan, Santosh, and Akshay for reviewing drafts of this and offering valuable suggesions. If you have any suggestions, happy to hear them

---

[passforios]: https://mssun.github.io/passforios/ “Open Source, no-network, minimalist pass client for iOS” [okc]: https://www.openkeychain.org/ [pass]: https://passwordstore.org [tile]: https://www.thetileapp.com/en-us/ [slim]: https://www.thetileapp.com/en-us/store/tiles/slim “I have the older version of the Tile Slim” [ssh-u2f]: https://www.undeadly.org/cgi?action=article;sid=20191115064850 “Does it really count as 2FA if both your SSH and U2F is the same device?” [nitrokey-start]: https://shop.nitrokey.com/shop/product/nitrokey-start-6

1. I moved away from Lastpass after Tavis Ormandy reported a RCE vulnerability on their browser extension. Their wikipedia page mentions 2 breaches, and 3 security incidents. It has never undergone a security audit (unlike bitwarden) and is not something I recommend anymore. ↩

2. The pass-tomb extension bypasses this limit and encrypts your filenames as well. ↩

3. I have my own git server configured as a fallback if it goes down. I ensure the same controls on my Git server as Gitea, and it runs in my living room. ↩

4. I lost my previous GPG key because my Yubikey stopped working ↩

5. The jury is still out on whether this counts as an “independent second factor”. ↩

6. The domain is stuck in a legal limbo, because of an ongoing case between my registrar and NIXI (which runs the .in registry). If you have any suggestions/ideas, please reach out. ↩

7. Namecheap announced U2F support in April 2019, and while it was buggy at first, it has definitely improved. ↩

8. The one major exception is lastpass, which I no longer recommend. ↩

]]> Nemo https://mrkaran.dev/posts/kubectl-wait/ https://mrkaran.dev/posts/kubectl-wait/ Wed, 01 Jan 2020 02:40:55 GMT . For eg, for deployment and pods: $ kc describe deployments/{deployment_name} | grep Conditions -A 5 Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available True MinimumReplicasAvailable $ kc describe pods/{pod_name} | grep Conditions -A 5 Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True So, now you’ll set the value for condition according to your choice. This will be pretty useful in CI/CD pipelines. That’s pretty much it. Unrelated, but I thought about doing more such short posts and be consistent with more of writing. If you liked the short and precise format or have any feedback on it, do reach out to me on Twitter. Happy New Year :) Fin!]]> For the longest time I’ve had these commands in my .gitlab-ci.yml file for a K8s CD pipeline:

    ...
    - kubectl apply -k overlays/prod
    - echo "Waiting for 15 seconds for pods to be restarted" && sleep 15
    - kubectl get po
    ...

So, basically I apply the changes to cluster using kubectl apply and wait for arbitary decided time (15 seconds) to see the pod status, hoping by that time the new deployments would have been active and old pods would be deleted. As the traditional SRE saying goes Hope is not a strategy this was clearly hacky and I knew it back then, just didn’t priortise enough to find a replacement. Recently got to know about kubectl wait and woah, this is exactly what I needed. I can wait till either the condition is true or a timeout happens, whichever is earlier. This is so much better than the previous hack.

kubectl wait --for=condition=available --timeout=60s --all deployments

Here the condition depends on the resource you are selecting. You can see the values for Conditions using kubectl describe <resource>. For eg, for deployment and pods:

$ kc describe deployments/{deployment_name} | grep Conditions -A 5

Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable

$ kc describe pods/{pod_name} | grep Conditions -A 5

Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True

So, now you’ll set the value for condition according to your choice. This will be pretty useful in CI/CD pipelines. That’s pretty much it.

---

Unrelated, but I thought about doing more such short posts and be consistent with more of writing. If you liked the short and precise format or have any feedback on it, do reach out to me on Twitter.

Happy New Year :)

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/introducing-kubekutr/ https://mrkaran.dev/posts/introducing-kubekutr/ Mon, 30 Dec 2019 02:40:55 GMT

kubekutr was born out of my frustration of organising K8s resource manifests files/directories. For the uninitiated, K8s lets you hold the state of your cluster declaratively as “manifest” files. K8s does so by a lot of asynchronous control loops to check whether the desired state matches the real world state and in case of drifts, it resets back to the user desired state (I oversimplified this, but hope you get the drift ;)). These files are predominantly YAML but there’s support for JSON as well. Anyway, to create these manifest files for a production level project is quite a bit of manual labour. The API spec of every resource in Kubernetes is quite daunting and overwhelming. There are tools like Helm which abstract away the complexity of these YAML with it’s own templating system. There are quite a lot of these charts available for 3rd party apps here. The idea is you populate the Chart with just your own config “values” and you’ve a deployment ready in no time. Admittedly this works quite well for something you want to take out for a quick spin but personally I am not quite a fan of hiding away the complexity with a magic layer. Also, the problem with Helm was the “Chart” (and templates) still had to be written by someone if you have a bespoke application. Helm is more geared towards common off the shelf apps like DBs, Key Value stores, web proxies etc.

I found out kustomize few months back and quite happy with it’s approach towards managing manifests. The basic idea behind kustomize is that you create a base and any kind of “customisations” must come as overlays. This is such a powerful technique over wrangling templates. kustomize A common approach is to name these overlays based on the environment. For example, dev deployment can have replicas: 1 for a pod, but prod can apply a “patch” to update with repliacs: 3. This way of separating two environments helps a lot when you follow GitOps approach of deployment. All fine and good, until I realised I spent way too much time on copy-pasting the bases for different projects and manually editing these files for the new project config.

Then I did what any other programmer would do, spend some more time to automate :P And that is how kubekutr was born. (Quite an anticlimax I know!)

kubekutr is a really simple tool to bootstrap a Kustomize base. kubekutr reads a config file, templates out different resources and produces them as YAML files. Now, I know a lot of you reading this would be going Another damn templating solution in your mind and while that reaction is warranted, given that we have 200+ tools in the community (everyone trying to solve similar problems in their own ways), I legit could not find a simple enough tool which would let the boring part of scaffolding a base out of my way and let me focus on what’s more important: the actual deployment. Hence I just decided to roll out my own solution which is the best one according to IKEA effect (just kidding).

Workflow#

So, let’s say you need to create a Nginx deployment, the kubekutr config.yml would look something like this:

deployments:
  - name: nginx
    replicas: 1
    labels:
      - name: 'service: nginx'
    containers:
      - name: nginx
        image: 'nginx:latest'
        portInt: 80
        portName: nginx-port
services:
  - name: nginx
    type: ClusterIP
    port: 80
    targetPort: 80
    labels:
      - name: 'service: nginx'
    selectors:
      - name: 'service: nginx'

To create the base:

kubekutr -c config.yml scaffold -o nginx-deployment

nginx-deployment folder is initialised and you can view deployments/nginx.yml and service/nginx.yml which kubekutr created.

$ tree nginx-deployment

|-- base
|   |-- deployments
|   |   `-- nginx.yml
|   |-- ingresses
|   |-- services
|   |   `-- nginx.yml
|   `-- statefulsets

$ cat base/deployments/nginx.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    service: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      service: nginx
  template:
    metadata:
      labels:
        service: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
          - containerPort: 80
            name: nginx-port

$ cat base/services/nginx.yml

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    service: nginx
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: ClusterIP
  selector:
    service: nginx

You can now use the generated folder as a Kustomize base.

Non Goals#

kubekutr isn’t meant to replace the existing tools, it’s just a real simple cookie cutter approach to kustomize bases and that’s pretty much it. kustomize is native to Kubernetes and exposes the full API spec to end users. I feel that is much more better approach than templating solutions, the users must be exposed to the standard conventions rather than a random tool’s own config fields. The benefits are the same conventions can then be used across a wide variety of tools (like kubekutr) and users are in better control of the underlying resources. Adding a layer of magic also makes it harder to debug when shit goes down. Hence kubekutr chose kustomize to do all the heavy lifting of managing manifests.

There’s a lot of scope of improvements, but I wanted to just Ship It! and get some initial feedback. Let me know your thoughts on this :)

Fin!

]]> Karan Sharma https://www.prashanthudupa.com/my-son-knows-how-to-create-his-relationships/ https://www.prashanthudupa.com/my-son-knows-how-to-create-his-relationships/ Wed, 04 Dec 2019 04:13:54 GMT Prashanth Udupa Philosophy Unschooling https://mrkaran.dev/posts/dig-overview/ https://mrkaran.dev/posts/dig-overview/ Mon, 11 Nov 2019 05:27:55 GMT > DiG 9.10.6 <<>> mrkaran.dev ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23292 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1220 ;; QUESTION SECTION: ;mrkaran.dev. IN A ;; ANSWER SECTION: mrkaran.dev. 60 IN A 206.189.89.118 ;; Query time: 6 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Tue Oct 29 23:13:31 IST 2019 ;; MSG SIZE rcvd: 67 This is the most basic example for dig. Let’s explore some of the additional options. Keep it short# dig +short keeps the information to bare minimum and only displays the ANSWER. dig +short mrkaran.dev 206.189.89.118 Nameserver details# If you want to find the Nameserver for your DNS records, you can use the query type ns. $ dig mrkaran.dev ns +short alec.ns.cloudflare.com. cruz.ns.cloudflare.com. ns is one of the many query types you can use to indicate which type of DNS record you want to fetch. Default is A record which returns the IPv4 address of the domain (unless it’s a root domain, in which case the default query type is NS). Some other examples of query types are mx, AAAA, TXT etc. Fun Fact: ANY query type has become obsolete as per the new RFC8482 and DNS operators can choose to not respond to this query. The reason for this is that the payload response size for an ANY query is quite huge (since it has to return all type of DNS records) and this could affect the performance of authoritative servers in case of a DNS amplification attack. Using different DNS server# Let’s say you want to switch to a different resolver, you can use @ followed by the address of your DNS server. $ dig mrkaran.dev @9.9.9.9 Reverse DNS Lookup# This one’s actually pretty cool. dig -x lets you query the IP and retrieve the hostname details for that IP. dig -x 206.189.89.118 Multiple queries# You can input a list of domain names and pass the file with the arg -f to dig. $ cat digfile mrkaran.dev joinmastodon.org zoho.com To list down all MX records for the domains in a file, you can use something like: $ dig -f digfile +noall mx +answer mrkaran.dev. 242 IN MX 10 mx.zoho.in. mrkaran.dev. 242 IN MX 20 mx2.zoho.in. mrkaran.dev. 242 IN MX 50 mx3.zoho.in. joinmastodon.org. 21599 IN MX 10 in1-smtp.messagingengine.com. joinmastodon.org. 21599 IN MX 20 in2-smtp.messagingengine.com. zoho.com. 299 IN MX 10 smtpin.zoho.com. zoho.com. 299 IN MX 20 smtpin2.zoho.com. zoho.com. 299 IN MX 50 smtpin3.zoho.com. Search List# I learnt this recently while debugging a DNS issue in one of the Kubernetes pods. Dig doesn’t use search paths by default, so if you have a service say redis inside a namespace dig won’t fetch any result: $ dig redis +short # empty output, indicates no record found This is because a service name in Kubernetes is of the form service.namespace.svc.cluster.local. So, we should actually be querying for redis.myns.svc.cluster.local and we’ll get our result. But isn’t that too long and painful (sorry for the pun) to type? So, there’s another option +search which can be used to find all domains matching the search path defined in /etc/resolv.conf namesever configurations. $ cat /etc/resolv.conf nameserver 10.100.0.10 search myns.svc.cluster.local svc.cluster.local cluster.local We can now query for redis with this search list: dig redis +search +short 10.100.32.73 DNSSec Validation# dig even lets you validate the DNS records you received using DNSSEC validation. $ dig mrkaran.dev +dnssec ; <<>> DiG 9.10.6 <<>> mrkaran.dev +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36275 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 1452 ;; QUESTION SECTION: ;mrkaran.dev. IN A ;; ANSWER SECTION: mrkaran.dev. 20 IN A 178.128.17.49 mrkaran.dev. 20 IN RRSIG A 13 2 20 20191112173050 20191110153050 34505 mrkaran.dev. Tl3zD6EqfVRvZi79ahePQcAXnbSUY9ZEYx/KwXnDUyonlrCKuBHzIYYC MJoVns410+sOwbIrcAdLgx+eiMYqRQ== ;; Query time: 65 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Mon Nov 11 22:01:01 IST 2019 ;; MSG SIZE rcvd: 163 The important bit to note here is the ad flag set which represents Authenticated Data. The records will only be returned if the validation succeeds (unless you also specify +cd which indicates Checking Disabled flag.) On a server which doesn’t have DNSSEC enabled, you can see no records are returned with the +dnssec flag. $ dig dnssec-failed.org +dnssec ; <<>> DiG 9.10.6 <<>> dnssec-failed.org +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 19886 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;dnssec-failed.org. IN A ;; Query time: 335 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Mon Nov 11 22:03:50 IST 2019 ;; MSG SIZE rcvd: 35 That pretty much broadly covers some practical examples with dig. I will soon write a detailed post on how DNSSEC validation works and why it needs to be mainstream. Fin!]]> Dig is a DNS lookup utility developed by BIND which helps a lot while troubleshooting DNS issues (which are more common than you probably think #hugops). I use dig fairly often and thought to write an introductory guide on how you can use dig with some practical examples that’ll help you dig through DNS issues faster (sorry for the lame pun, couldn’t resist.)

Basics#

The most basic and common usage for dig is to query the authorative servers for a particular domain and retrieve the IP. If it’s an IPv4 then you should be looking at A record, while if it’s IPv6 then AAAA record is your friend. Let’s see the DNS records for the site you’re currently on:

➜  ~ dig mrkaran.dev

; <<>> DiG 9.10.6 <<>> mrkaran.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23292
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
;; QUESTION SECTION:
;mrkaran.dev.			IN	A

;; ANSWER SECTION:
mrkaran.dev.		60	IN	A	206.189.89.118

;; Query time: 6 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Oct 29 23:13:31 IST 2019
;; MSG SIZE  rcvd: 67

This is the most basic example for dig. Let’s explore some of the additional options.

Keep it short#

dig +short keeps the information to bare minimum and only displays the ANSWER.

dig +short mrkaran.dev
206.189.89.118

Nameserver details#

If you want to find the Nameserver for your DNS records, you can use the query type ns.

$ dig mrkaran.dev ns +short
alec.ns.cloudflare.com.
cruz.ns.cloudflare.com.

ns is one of the many query types you can use to indicate which type of DNS record you want to fetch. Default is A record which returns the IPv4 address of the domain (unless it’s a root domain, in which case the default query type is NS). Some other examples of query types are mx, AAAA, TXT etc.

Fun Fact: ANY query type has become obsolete as per the new RFC8482 and DNS operators can choose to not respond to this query. The reason for this is that the payload response size for an ANY query is quite huge (since it has to return all type of DNS records) and this could affect the performance of authoritative servers in case of a DNS amplification attack.

Using different DNS server#

Let’s say you want to switch to a different resolver, you can use @ followed by the address of your DNS server.

$ dig mrkaran.dev @9.9.9.9

Reverse DNS Lookup#

This one’s actually pretty cool. dig -x lets you query the IP and retrieve the hostname details for that IP.

dig -x 206.189.89.118

Multiple queries#

You can input a list of domain names and pass the file with the arg -f to dig.

$ cat digfile
mrkaran.dev
joinmastodon.org
zoho.com

To list down all MX records for the domains in a file, you can use something like:

$ dig -f digfile +noall mx +answer
mrkaran.dev.		242	IN	MX	10 mx.zoho.in.
mrkaran.dev.		242	IN	MX	20 mx2.zoho.in.
mrkaran.dev.		242	IN	MX	50 mx3.zoho.in.
joinmastodon.org.	21599	IN	MX	10 in1-smtp.messagingengine.com.
joinmastodon.org.	21599	IN	MX	20 in2-smtp.messagingengine.com.
zoho.com.		299	IN	MX	10 smtpin.zoho.com.
zoho.com.		299	IN	MX	20 smtpin2.zoho.com.
zoho.com.		299	IN	MX	50 smtpin3.zoho.com.

Search List#

I learnt this recently while debugging a DNS issue in one of the Kubernetes pods. Dig doesn’t use search paths by default, so if you have a service say redis inside a namespace dig won’t fetch any result:

$ dig redis +short
# empty output, indicates no record found

This is because a service name in Kubernetes is of the form service.namespace.svc.cluster.local. So, we should actually be querying for redis.myns.svc.cluster.local and we’ll get our result. But isn’t that too long and painful (sorry for the pun) to type?

So, there’s another option +search which can be used to find all domains matching the search path defined in /etc/resolv.conf namesever configurations.

$ cat /etc/resolv.conf
nameserver 10.100.0.10
search myns.svc.cluster.local svc.cluster.local cluster.local

We can now query for redis with this search list:

dig redis +search +short
10.100.32.73

DNSSec Validation#

dig even lets you validate the DNS records you received using DNSSEC validation.

$ dig mrkaran.dev +dnssec
; <<>> DiG 9.10.6 <<>> mrkaran.dev +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36275
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1452
;; QUESTION SECTION:
;mrkaran.dev.			IN	A

;; ANSWER SECTION:
mrkaran.dev.		20	IN	A	178.128.17.49
mrkaran.dev.		20	IN	RRSIG	A 13 2 20 20191112173050 20191110153050 34505 mrkaran.dev. Tl3zD6EqfVRvZi79ahePQcAXnbSUY9ZEYx/KwXnDUyonlrCKuBHzIYYC MJoVns410+sOwbIrcAdLgx+eiMYqRQ==

;; Query time: 65 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Nov 11 22:01:01 IST 2019
;; MSG SIZE  rcvd: 163

The important bit to note here is the ad flag set which represents Authenticated Data. The records will only be returned if the validation succeeds (unless you also specify +cd which indicates Checking Disabled flag.)

On a server which doesn’t have DNSSEC enabled, you can see no records are returned with the +dnssec flag.

$ dig dnssec-failed.org +dnssec
; <<>> DiG 9.10.6 <<>> dnssec-failed.org +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 19886
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;dnssec-failed.org.		IN	A

;; Query time: 335 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Nov 11 22:03:50 IST 2019
;; MSG SIZE  rcvd: 35

That pretty much broadly covers some practical examples with dig. I will soon write a detailed post on how DNSSEC validation works and why it needs to be mainstream.

Fin!

]]> Karan Sharma https://mrkaran.dev/posts/gitops-kubernetes/ https://mrkaran.dev/posts/gitops-kubernetes/ Mon, 11 Nov 2019 05:27:55 GMT In this post, I’d like to share my experience and learnings about configuring a deployment pipeline for Kubernetes. I’ll be using Gitlab CI/CD and AWS EKS to demonstrate the concept, but the core idea remains the same: all changes must come declaratively from a single source of truth. GitOps is a relatively newer term in the town but goes back to the fundamentals of Infra as Code.

GitOps fundamentally is an operating model to perform tasks on Kubernetes related to deployments, configuration, secrets and monitoring workloads. All kind of changes must be performed via a single place, which happens to be a git repo. Benefits of that are what basically benefits of version controlling the code is. So why treat infra as any different? git happens to be the single source of truth for your infra, rollbacks are easy as reverting to last known good configuration and every change can be observed/verified.

Goals#

A lot of tutorials/blog posts hitherto cover a very basic scenario where they do kubectl apply and voila the deployment’s live. However we all know things are very different (to say the least) in production, so this post will cover all aspects of deployment:

• Creating Manifests
• Environment Promotion
• Handling config and secrets
• Authorization of CI/CD in the cluster

Basics#

A GitOps workflow looks like:

Push vs Pull#

There are 2 approaches to how you can handle deployments to a cluster. In a Pull based approach, the cluster runs a synch controller program which continmioously syncs the state of cluster with a Git repo. Any changes you make to the Git repo will be synced automatically in cluster. The idea is that there should be no drift in the desired state via Git repo and the actual state of cluster. Flux, Argo are good tools if you want a Pull based pipeline. The merits of Pull based pipeline is it’s more secure, since the deployment is actually happening inside cluster and no external sytstem needs to communicate to your production infra. The demerits are sometimes you’ve to wait for the changes to be synced (every controller runs these sync process in a loop with a sleep which can be configured). Also using any kind of preprocessing tools like Kustomize becomes difficult, since Flux just syncs the state and applies those changes. Handling of secrets is yet another concern, you need to look . And finally GitOps is a relatively newer tech in market so GitOps tooling is still nascent and like with any other relatively (non battle tested) software you’re gonna find bugs.

Push Approach however is a traditional CD approach, where the CD server talks to the cluster and applies changes through commands. In context of normal EC2 deployments, those commands could be SSH into server, running ansible playbook etc. In context of K8s however kubectl does the magic for us. The CD server needs to talk to the K8s API server and run kubectl commands to change the cluster state. The merits of this approach are you can run all sorts of commands inside deployment pipeline and make it fully customisable. Handling of secrets also can be handled natively (like Gitlab env variables) or encrypted in git. The demerits is that your production cluster is now exposed to your CD server.

Overall, if you have an airgapped CD server with no inbound ports open, access controll the user auth to CD, I found the Push approach to be more preferable. YMMV.

Writing the pipeline#

I’ve created a docker image eks-gitops which I’ll be using throughout the pipeline. This container image contains popular tools like kustomize, kubeval etc and scripts to configure access to cluster using kubectl using aws-iam-authenticator. I’ve written more about how RBAC works inside EKS here.

Excerpt from .gitla-ci.yml:

# Use this as base image for all jobs unless overriden
default:
  image:
    name: mrkaran/eks-gitops:latest
    entrypoint: ["/bin/sh", "-c"]
### Pipeline
stages:
  - validate
  - deploy

Prepare the manifests#

I use Kustomize to prepare the manifests. Advantage of Kustomize is writing template free YAMLs but still be able to customise them heavily using overlays. For different environments, you can apply certain changes like increasing resource requests, adding more storage, while keeping the base same.

Here’s a folder structure (from a real GitOps repo) I follow for manifests:

.
├── base
│   ├── deployments
│   │   ├── app.yml
│   │   ├── celery.yml
│   │   └── nginx.yml
│   ├── ingresses
│   │   └── web.yml
│   ├── kustomization.yaml
│   ├── services
│   │   ├── app.yml
│   │   ├── nginx.yml
│   │   └── redis-headless.yml
│   ├── statefulsets
│   │   └── redis.yml
│   └── volumes
│       └── redis.yml
├── kubekutter.yml
├── Makefile
├── overlays
│   ├── dev
│   └── prod
│       ├── configs
│       │   ├── app-config.env
│       │   └── app-nginx.conf
│       ├── kustomization.env.yml
│       ├── namespace.yml
│       ├── patches
│       │   ├── configure-configmap-volume.yml
│       │   ├── modify-alb.yml
│       │   └── resource-limits.yml
│       └── rbac.yml
└── README.md

Shameless Plug: I created kubekutr which makes managing of these manifests using kustomize a breeze.

Some things to note here:

• Inside base/, I keep all the base resources required for the app to run. The resources can be Service, Deployment, Ingress etc.
• Inside overlays there are multiple folders for different environment. This is very crucial as we want to separate the production config with a UAT config. Last-mile configuration to the base becomes very easy with this folder structure, since you only now need to build the manifests by targetting a specific folder in CI.
• Inside overlays/{env}/patches are all the “patches” you want to do to the base resource. Think like replica count, ALB subnets (since different env can be in different VPCs), increasing resource limits and stuff like that.
• rbac.yml abd namespace.yml is the only missing piece because it’s like a chicken and egg problem. I cannot deploy directly (at first go) from a CI/CD if I don’t have a namespace created since the CD server is configured only has limited namespaced restricted access. So unless I create a namespace, add proper RBAC for the CD server I cannot do any deployments from CD. Note however this is only a first time step, which I guess is okay.

Lint yo manifest#

I’m using kubeval to lint the manifests. The manifests have to prepared by kustomize. CI_ENVIRONMENT_NAME is set by Gitlab when you specify an environment for a job. Don’t sweat about this part, I’ll describe it more as we proceed.

# Validate the yaml using kubeval
.lint:
  extends: .prepare-manifest
  stage: validate
  script:
    - echo "Linting manifest for ${CI_ENVIRONMENT_NAME}"
    - kustomize build overlays/$CI_ENVIRONMENT_NAME --load_restrictor none | kubeval

Setup environment#

You can define environment name as:

# Create an environment to record all jobs for this env
.prod: &prod
  environment:
    name: prod
    url: https://prod.site
.dev: &dev
  environment:
    name: dev
    url: https://dev.site

Any job which is to be executed in a particular environment can include this variable and CI_ENVIRONMENT_NAME will be automatically set.

A cool feature of Gitlab is that you can restrict Variables scoped to the environment they are defined in.

Configure Secrets#

All secrets are defined as Environment Variables in Gitlab CD pipeline. While running the job, the runner has access to these variables and with the help of secretGenerator in kustomize, the Secret is created.

I use secretGenerator because any time a K8s secret changes, kustomize appends with a new suffix, which makes the Replication Controller believe that they deployment has changed. So a new deployment is automatically triggered.

Authenticate to cluster#

EKS uses aws-iam-authenticator and uses IAM access roles to allow the cluster to perform actions. Since this is a push based pipeline, you need to allow the access from your CD server to port 443.

Deploy changes#

This is as simple as kubectl apply which configures all the changes and diff between cluster and real world state.

Here’s a full gitops repo if you’re interested in checking it out:

]]> Karan Sharma https://mrkaran.dev/posts/intro-rbac-kubernetes/ https://mrkaran.dev/posts/intro-rbac-kubernetes/ Fri, 01 Nov 2019 12:40:55 GMT ... mapUsers: | - userarn: arn:aws:iam:::user/ username: 10xdevs-fullaccess # RoleBinding name created in previous step groups: - 10xdevs # Group name created in previous step Verify it yourself# kubectl auth can-i create pods --all-namespaces: should fail kubectl auth can-i create pods -n coolapp: should work Fin!]]> EKS uses a custom authenticator tool called “aws-iam-authenticator”. The basic idea is to make the auth flow in EKS easier by using the tools you already use in AWS.

To wrap your head around the flow, consider three separate entities:

• A kubernetes resource (entire namespace, specific pods/configs etc)
• An action (get, watch, list, create, delete etc)
• An IAM role/user created on AWS

Your usecase might be to give an IAM role access to a Kubernetes namespace for example with certain restricted actions.

Since K8s is basically client-server communication with the API server, we must need to perform the following two things for every single API request which goes to the control plane:

• Authentication

kubectl talks to the API server using the token generated by aws-iam-authenticator. The API server passes on this info to the AWS servers which validated if the originating call is coming from a valid IAM user or not. If not, the access is denied from K8s.

• Authorization

Once the IAM user is validated, the IAM user needs to be mapped to a user to perform Authorization so K8s API server can know whether the action/resource requested for is to be allowed or not. This is where the aws-auth-cm.yml comes into the picture. It is basically a map of all IAM users with internal K8s groups or users created. The roles are assosciated to these users/groups so once the mapping is done, K8s API server can know what to do with the API request.

How does EKS know my IAM?#

If you configured your KUBECONFIG correctly using aws eks update-kubeconfig then you’ll find the below lines in your config file. This basically runs a aws cli command to find your IAM user/role configured on your system to produce a token.

   exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - eks-zero-public
      command: aws-iam-authenticator

Show me the YAML already#

• To create a Role and RoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: coolapp
  name: fullaccess
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["apps"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["batch"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["extensions"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["autoscaling"]
  resources: ["*"]
  verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: 10xdevs-fullaccess
  namespace: coolapp
subjects:
- kind: Group
  name: 10xdevs
  namespace: coolapp
roleRef:
  kind: Role
  name: fullaccess
  apiGroup: rbac.authorization.k8s.io

So, for all the normal folks who don’t grok YAML as fast as 10x devops engineers, I am basically creating a Role fullaccess with all permissions for a namespace coolapp (innovative, ikr). Then I am binding this Role to a Group called 10xdevs so that the group is allocated the role which has permissions. We will use this group 10xdevs to map our AWS user now.

• To create the map of IAM Role/User ARN with the above Role

Edit your aws-auth-cm.yml and add the below stuff:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles ... <skipping> ...

  mapUsers: |
    - userarn: arn:aws:iam::<account-id>:user/<user-name>
      username: 10xdevs-fullaccess # RoleBinding name created in previous step
      groups:
        - 10xdevs # Group name created in previous step

Verify it yourself#

• kubectl auth can-i create pods --all-namespaces: should fail

• kubectl auth can-i create pods -n coolapp: should work

Fin!

]]> Karan Sharma https://www.prashanthudupa.com/how-dare-you-because-obviously/ https://www.prashanthudupa.com/how-dare-you-because-obviously/ Thu, 03 Oct 2019 19:55:20 GMT Prashanth Udupa Moments https://mrkaran.dev/posts/home-server-setup/ https://mrkaran.dev/posts/home-server-setup/ Sun, 22 Sep 2019 02:40:55 GMT }} Setting up RPi# I downloaded Raspbian Buster Lite because it’s the easiest to setup. Next step is to flash the SD card and for that, I used Etcher. To enable SSH access, you need to create an empty file ssh in the root volume. sudo touch /boot/ssh Once all sorted, we can use Ansible to set up the basic OS stuff, like changing the default password, enabling password-less SSH login, timezone & locale settings, changing hostname etc. I’ll be sharing relevant Ansible snippets, if interested you can check out the complete playbook at mr-karan/hydra repo. We need to enable container features on the RPi so that containerd can run. Containers like Docker make use of cgroups (Linux kernel feature) which allows them to put resource limits on container processes like CPU and Memory. To enable cgroups, you need to edit /boot/cmdline.txt. - name: Add cgroup directives to boot command line config lineinfile: path: /boot/cmdline.txt regexp: '((.)+?)(\scgroup_\w+=\w+)*$' line: '\1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory' backrefs: yes Quick Tip: If you’re going to use RPi as a headless server (not connecting with any monitor) you can reduce the GPU Memory to lowest possible (16M) - name: Set GPU memory split to 16 MB lineinfile: path: /boot/config.txt line: "gpu_mem=16" create: yes Deploy K3s cluster# Next, we’ll come to the actual stuff, where we’ll download K3s binary and run as a systemd service. There’s a handy shell script to bootstrap the cluster provided by the Rancher team which setups the whole thing in one command. But if you wanna learn/play around I’d recommend you do things the hard way (it’s not all that hard tho. ((twss!)). - name: Download k3s binary armhf get_url: url: https://github.com/rancher/k3s/releases/download/{{ k3s_version }}/k3s-armhf dest: /usr/local/bin/k3s owner: root group: root mode: 755 when: ( ansible_facts.architecture is search("arm") ) and ( ansible_facts.userspace_bits == "32" ) On the worker node, the process is similar, except you have to run the K3s with agent compared to server argument in the control plane. Things get a bit interesting here though. You need to give the cluster server URL along with it’s token in the command. A unique token is generated by the server at (/var/lib/rancher/k3s/server/node-token) which is used to join the worker nodes. I did a bit of google-fu and got to know about this neat little Ansible module set_fact which lets you “store” a variable from one host and use it in a second host. Every Ansible host maintains a Python dict of “host facts”. In the second node, I access the cluster’s host fact dict, fetch the variable and use it in its systemd service template. Neat, ain’t it? Ansible has so many modules, it is mind-boggling. Reading and storing the variable as a “host” fact: # on the cluster - name: Read node-token from control node slurp: src: /var/lib/rancher/k3s/server/node-token register: node_token - name: Store control node-token set_fact: k3s_cluster_token: "{{ node_token.content | b64decode | regex_replace('\n', '') }}" Using the variable from the server host, in a template on the agent host: # on the agent (vars.yml) k3s_server_address: "{{ hostvars[groups['control'][0]].k3s_server_address }}" k3s_cluster_token: "{{ hostvars[groups['control'][0]].k3s_cluster_token }}" # use the value in a template ... [Service] ExecStart=/usr/local/bin/k3s agent --server {{ k3s_server_address }} --token {{ k3s_cluster_token }} ... P.S. Shoutout to Ansible tho. It is one of my fav infra tooling available out there. It has some gotchas that you need to be aware of but by and large, the experience has been quite pleasant. On the cluster, you should be able to see the nodes. Team #SelfHost# I am planning to host Bitwarden, Gitea and a Nextcloud instance on this cluster. Also will be using this as a testbed to play around with K8s internals. Stay tuned as I explore more of this! Cheers! :)]]> So, I got hold of 2 Raspberry Pi4 (still limited stocks in India) recently and wanted to build a Kubernetes cluster. Don’t ask why cause that would be pointless. I’ve little experience with a managed Kubernetes workload (Amazon EKS, which btw deserves its post 😝) but never really played around any of the K8s internal stuff yet. In this post, I’ll show you how I got a lightweight Kubernetes distro: K3s up and running.

k3s is a pretty great Kubernetes distro which passes the K8s Conformance tests. On ARM architectures, you’re pretty much resource-bounded and you want the resource footprint of your infra to be as minimal as possible. In k3s there are a few changes such as the persistent layer of K8s is replaced with SQLite instead of etcd, unusable (legacy/alpha) features of K8s are removed and cloud-provider plugins are not bundled (but can be installed separately). All of this together means just 40MB of a binary to run the cluster and ~250MB of memory usage on an idle cluster. Awesome, team Rancher :)

Automation is the key and even though I have just 2 nodes on RPi, I Ansible-ized the setup which I am hoping would save time in future if I add more nodes to the setup.

Hardware#

• 1x RPi4 4GB and 1x RPi4 2GB variant
• 2x Samsung EVO micro SD Card
• 2x USBC Cables
• 1x Amker Power Port (don’t compromise on the power supply, give enough juice so RPi doesn’t throttle)
• 1x TP-Link Network Switch (my router has only 1 usable LAN port)
• 2x CAT5 LAN cables (keeping it basic, you can get fancy flat LAN cables if you wish to)

This is how the final setup looks like:

{{< tweet 1175659256764219392 >}}

Setting up RPi#

I downloaded Raspbian Buster Lite because it’s the easiest to setup. Next step is to flash the SD card and for that, I used Etcher.

To enable SSH access, you need to create an empty file ssh in the root volume.

sudo touch /boot/ssh

Once all sorted, we can use Ansible to set up the basic OS stuff, like changing the default password, enabling password-less SSH login, timezone & locale settings, changing hostname etc. I’ll be sharing relevant Ansible snippets, if interested you can check out the complete playbook at mr-karan/hydra repo.

We need to enable container features on the RPi so that containerd can run. Containers like Docker make use of cgroups (Linux kernel feature) which allows them to put resource limits on container processes like CPU and Memory. To enable cgroups, you need to edit /boot/cmdline.txt.

- name: Add cgroup directives to boot command line config
  lineinfile:
    path: /boot/cmdline.txt
    regexp: '((.)+?)(\scgroup_\w+=\w+)*$'
    line: '\1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory'
    backrefs: yes

Quick Tip: If you’re going to use RPi as a headless server (not connecting with any monitor) you can reduce the GPU Memory to lowest possible (16M)

- name: Set GPU memory split to 16 MB
  lineinfile:
    path: /boot/config.txt
    line: "gpu_mem=16"
    create: yes

Deploy K3s cluster#

Next, we’ll come to the actual stuff, where we’ll download K3s binary and run as a systemd service. There’s a handy shell script to bootstrap the cluster provided by the Rancher team which setups the whole thing in one command. But if you wanna learn/play around I’d recommend you do things the hard way (it’s not all that hard tho. ((twss!)).

- name: Download k3s binary armhf
  get_url:
    url: https://github.com/rancher/k3s/releases/download/{{ k3s_version }}/k3s-armhf
    dest: /usr/local/bin/k3s
    owner: root
    group: root
    mode: 755
  when: ( ansible_facts.architecture is search("arm") )
    and
    ( ansible_facts.userspace_bits == "32" )

On the worker node, the process is similar, except you have to run the K3s with agent compared to server argument in the control plane. Things get a bit interesting here though. You need to give the cluster server URL along with it’s token in the command. A unique token is generated by the server at (/var/lib/rancher/k3s/server/node-token) which is used to join the worker nodes.

I did a bit of google-fu and got to know about this neat little Ansible module set_fact which lets you “store” a variable from one host and use it in a second host. Every Ansible host maintains a Python dict of “host facts”. In the second node, I access the cluster’s host fact dict, fetch the variable and use it in its systemd service template. Neat, ain’t it? Ansible has so many modules, it is mind-boggling.

Reading and storing the variable as a “host” fact:

# on the cluster
- name: Read node-token from control node
  slurp:
    src: /var/lib/rancher/k3s/server/node-token
  register: node_token

- name: Store control node-token
  set_fact:
    k3s_cluster_token: "{{ node_token.content | b64decode | regex_replace('\n', '') }}"

Using the variable from the server host, in a template on the agent host:

# on the agent (vars.yml)
k3s_server_address: "{{ hostvars[groups['control'][0]].k3s_server_address }}"
k3s_cluster_token: "{{ hostvars[groups['control'][0]].k3s_cluster_token }}"
# use the value in a template
...
[Service]
ExecStart=/usr/local/bin/k3s agent --server {{ k3s_server_address }} --token {{ k3s_cluster_token }}
...

P.S. Shoutout to Ansible tho. It is one of my fav infra tooling available out there. It has some gotchas that you need to be aware of but by and large, the experience has been quite pleasant.

On the cluster, you should be able to see the nodes.

Team #SelfHost#

I am planning to host Bitwarden, Gitea and a Nextcloud instance on this cluster. Also will be using this as a testbed to play around with K8s internals. Stay tuned as I explore more of this!

Cheers! :)

]]> Karan Sharma https://kcsrk.info/ocaml/multicore/job/2019/09/16/1115-multicore-job/ https://kcsrk.info/ocaml/multicore/job/2019/09/16/1115-multicore-job/ Mon, 16 Sep 2019 11:59:00 GMT Multiple Research Software Engineer positions are available in the Department of Computer Science and Engineering at the Indian Institute of Technology, Madras to develop Multicore OCaml and enable Tezos ecosystem to benefit from Multicore OCaml.

Background

The Multicore OCaml project aims to add native support for scalable concurrency and shared memory parallelism in OCaml. At its core, Multicore OCaml extends the OCaml programming language with effect handlers for expressing scalable concurrency and a high-performance concurrent garbage collector aimed at responsive networked applications. Multicore OCaml is also the first industrial-strength language to come equipped with an efficient yet modular memory model, allowing high-level local program reasoning while retaining performance. Multicore OCaml is actively being developed and core features are being upstreamed to OCaml.

Tezos is an open-source smart contract platform for decentralized applications and assets. Tezos uses a self-amending cryptographic ledger - It achieves consensus not just about the state of a ledger, but about the state of its own protocol. The primary protocol of Tezos utilizes proof of stake and supports Turing complete smart contracts in a domain-specific language called Michelson. Tezos codebase is written in OCaml and extensively uses OCaml ecosystem libraries and tools such as Lwt, OPAM, and Irmin.

Roles

There are two roles:

• Compiler Engineer: Runtime system improvements to the OCaml programming language in order to make it compatible with multicore support. Implementing new features in Multicore OCaml compiler.
• Application Engineer: Developing core OCaml libraries that take advantage of multicore support. Adding parallelism support for Tezos ecosystem libraries and tools such as Lwt, Irmin, and dune.

Positions

The positions available are:

Role	Minimum Qualification	Pay Range (Per Month)
Project Engineer	BE / BTech / Master’s in Science/MCA or equivalent	Rs.21,500 to Rs.75,000
Senior Project Engineer	ME / MTech (or) BE / BTech / Master’s in Science / MCA or equivalent with 2 years experience	Rs.27,500 to Rs.1,00,000
Senior Project Officer / Post-doctoral Researcher	Ph.D. in Engineering or Sciences (or) ME / MTech with 3 years experience (or) BE / BTech / Master’s in Science / MCA or equivalent with 5 years experience	Rs.35,000 to Rs.1,50,000
Principal Project Officer	Ph.D. in Engineering or Sciences with 7 years experience (or) ME / MTech with 10 years experience (or) BE / BTech / Master’s in Science / MCA or equivalent with 12 years experience	Rs. 48,000 to Rs.2,25,000

The appointment will be made for 6 months initially and can be extended up to 2 years. The project engineers have the option of enrolling in the MS program at CSE, IIT Madras after 6 months. Such candidates may appear to the interview directly, without having to write GATE.

You will work closely with the OCaml Labs group, University of Cambridge, UK and Tarides, France. All of the work done will be made available as liberally licensed open-source software.

Skills

Compiler Engineer

Necessary:

• Excellent working knowledge of C, concurrent and parallel programming
• Knowledge of compilers (not necessarily of functional programming languages), operating systems, x86 & ARM assembly programming

Desired:

• Experience developing and/or maintaining performant software systems
• Experience with a functional programming language such as Haskell, OCaml, Scala, Scheme, Elm, or Elixir.
• Track record of open source contributions.
• Understanding of benchmarking techniques and analyzing results

Applications Engineer

Necessary

• Excellent working knowledge of operating systems, concurrent and parallel programming
• Experience with a functional programming language such as Haskell, OCaml, Scala, Scheme, Elm, or Elixir.

Desired

• Track record of contributions to large open-source software systems
• Understanding of benchmarking techniques and analyzing results

Apply

Write to kcsrk@iitm.ac.in with the subject “IITM Multicore OCaml 2019: Compiler Engineer” or “IITM Multicore OCaml 2019: Application Engineer” based on the role to express interest. Please include:

1. Curriculum Vitae
2. A summary of your experience in relevant technologies and software
3. Any open source contributions

]]> KC Sivaramakrishnan https://www.prashanthudupa.com/swayam-a-chance-to-meet-alt-schoolers/ https://www.prashanthudupa.com/swayam-a-chance-to-meet-alt-schoolers/ Wed, 28 Aug 2019 12:22:15 GMT Prashanth Udupa Unschooling https://captnemo.in/blog/2019/08/11/ipad-downgrade-ios-6-8/ https://captnemo.in/blog/2019/08/11/ipad-downgrade-ios-6-8/ Sun, 11 Aug 2019 00:00:00 GMT Update: Apple is no longer signing iOS6 for the iPad 2, so this is no longer feasible.

I own a iPad 2 (GSM), which is rarely used these days because it is too slow with the latest iOS 9 upgrades. It is a 8 year old device, but I can’t just install Linux on it and make it usable, which is what I do with most other devices.

The next best thing was to downgrade the iOS version. The device is anyway un-supported at this point, so I might as well go there. Apple has restrictions on which iOS releases are installable at any point on any device, ~but thankfully they are still signing iOS6 for my device for some legal reasons~.

Steps:

1. Download the iOS firmware for your device from https://ipsw.me/#platform
2. Launch iTunes
3. Option+Click on the Restore button in iTunes
4. Select the file you just downloaded.
5. Disable iCloud on the device
6. Upgrade to iOS8

This is possible as long as Apple is signing the IPSW for your device. The case of iOS6 being signed seems to be true for:

• iPhone 4S
• iPad 2

Security Notes

Running an unsupported OS is not something I take lightly. Here’s a list of defensive measures I took to ensure that I’m not at risk while doing so:

1. Try to keep the device always in Airplane mode.
2. Keep sensitive data off the device. No photos/keychain sync for eg. Don’t enable Calendar/Contact/Media sync.
3. Enable Restrictions on the device:
   • Restrict Safari to limited websites.
   • Disable application installs.
   • Disable iTunes store/iBooks store etc.
   • Disable GPS/Bluetooth.
   • Limit Background Refresh to very few trusted applications.
4. Limit number of applications (I only have Kybooks installed).
5. Disable Javascript on Safari.

If possible, I’d recommend using a separate Apple Account on the device.

]]> Nemo https://www.prashanthudupa.com/from-nandinis-travel-diary/ https://www.prashanthudupa.com/from-nandinis-travel-diary/ Tue, 30 Jul 2019 05:13:18 GMT Prashanth Udupa Unschooling https://mrkaran.dev/posts/personal-networking-setup/ https://mrkaran.dev/posts/personal-networking-setup/ Sat, 29 Jun 2019 12:40:55 GMT server_public_key $ ls server_private_key server_public_key Configuring Wireguard (server)# $ touch /etc/wireguard/wg0.cong $ vim /etc/wireguard/wg0.conf # Add the following lines and modify the values [Interface] # Configuration settings for a separate network interface Address = 10.200.200.1/24 # You can choose any private subnet SaveConfig = false # Wireguard can configure additional peers automatically without reloading wireguard, for some reason this didn't work well for me PrivateKey = # Output of the `server_private_key` generated in the above setup ListenPort = 51820 # Default port # Add the peers (clients which connect to the wireguard server) [Peer] # MBP PublicKey = AllowedIPs = 10.200.200.2/32 [Peer] # Android PublicKey = AllowedIPs = 10.200.200.3/32 Configuring Wireguard (client)# Repeat the step of generating a Public/Private key pair. You can also take a look at Subspace which is a nice GUI tool which helps you create additional profiles for your devices where generating key/pair is not convenient like mobile phones. I didn’t get the time to set it up personally, so I generated public/private keys from my laptop itself and then configured it manually using an Android app Viscerion which is a wireguard client app. For the client, your config file should look like: [Interface] PrivateKey = Address = 10.200.200.2/32 [Peer] PublicKey = AllowedIPs = 0.0.0.0/0 Endpoint = :51820 PersistentKeepalive = 25 Recap# If you got overwhelmed at this point, let’s recap what just happened. We have 2 config files, one for the server and one for the client which happens to be my laptop. On each of the devices, generate a public/private key pair. On the server side, while configuring peer, give the public key of the client. On the client side, while configuring peer,give the public key of the server. This is similar to how ssh works. Wireguard uses Curve25519 crypto technique to generate a public/private key pair, which honestly looks so better than lengthy ECDSA/RSA ones :P Now let us start the wireguard service. wireguard provides a nice wrapper wg-quick which does the following things when you start: sudo wg-quick up wg0 [#] ip link add wg0 type wireguard [#] wg setconf wg0 /dev/fd/63 [#] ip -4 address add 10.200.200.1/24 dev wg0 [#] ip link set mtu 1420 up dev wg0 You can verify a new network interface now by: $ ip addr | grep wg0 5: wg0: mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000 inet 10.200.200.1/24 scope global wg0 You can also view the wireguard connection status by: $ sudo wg show interface: wg0 public key: private key: (hidden) listening port: 51820 peer: allowed ips: 10.200.200.2/32 Almost There, But Not Quite# Turn on your wireguard client and you will notice a strange thing. At this point, you’re not able to browse the internet but you are able to connect to the wireguard server and even wireguard is acknowledging that (notice the last two lines in the following snippet): $ sudo wg show interface: wg0 public key: private key: (hidden) listening port: 51820 peer: endpoint: :64882 allowed ips: 10.200.200.2/32 *latest handshake: 8 seconds ago* *transfer: 754.60 KiB received, 5.59 MiB sent* Wireguard status shows that our client can reach the wireguard server. But still, we’re unable to browse the internet on our client. To debug this further, let’s use tcpdump and monitor the packets coming in wg0 interface. Since tcpdump’s output can be overwhelming and contains a lot of noise, let’s filter to monitor only the ICMP packets. So we’ll be using ping from a client which is the easiest way to send ICMP packets from point A to point B. # on wireguard client ➜ ~ ping $(curl icanhazip.com) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14 100 14 0 0 16 0 --:--:-- --:--:-- --:--:-- 16 PING (): 56 data bytes 64 bytes from : icmp_seq=0 ttl=64 time=112.126 ms 64 bytes from : icmp_seq=1 ttl=64 time=142.980 ms ^C --- ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss # on wireguard server $ sudo tcpdump -nni wg0 -Q in icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes 14:26:31.991498 IP 10.200.200.2 > : ICMP echo request, id 52630, seq 0, length 64 14:26:33.027288 IP 10.200.200.2 > : ICMP echo request, id 52630, seq 1, length 64 Okay, this is getting interesting. wg0 is definitely receiving packets from our client. Arghhh. Now does it strike to you? Our actual network routing is through eth0 (on DO usually) and wg0 is just some interface created by Wireguard. They have no “connection” (no pun intended) between them. Let us confirm if this actually is the problem by detecting ICMP traffic on this interface(eth0) using the same command: # on wireguard server $ sudo tcpdump -nni eth0 -Q in icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type RAW (Raw IP), capture size 262144 bytes ... # nothing happens even if we are sending PING from client Wow, see? No ICMP packets received on eth0. So that indeed is the problem and we have zero’ed it down using tcpdump. (Note to self: Learn more Linux debugging utils, these things are a godsend!) In order to fix this, we need to do 2 things: IP Forwarding $ vim /etc/sysctl.conf # check for this line and replace the value from 0 to 1 net.ipv4.ip_forward = 1 $ sudo sysctl -p IP Tables Rules We need to set up NAT between eth0(could be different for you) and wg0. This can be done using iptables and wireguard actually has a nice mechanism to run custom commands using PostUp/PostDown signals. # on wireguard server $ vim /etc/wireguard/wg0.conf # add these lines in [Interface] section PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE # Configure iptables to setup a NAT on eth0 and forward the packets (ipv4 and ipv6) on interface wg0 to eth0 PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE # Delete the rule since when wireguard is down, wg0 doesn't exist Now everything’s set up and we are browsing the internet privately using a VPN. Ensure that your public IP is of the VPN server when you’re browsing and it’s not leaking. Setting up Pi-hole# Installing Pi-hole is as simple as curl -sSL https://install.pi-hole.net | bash You can read more about installation in the official docs. A nice GUI is always a plus, so make sure you enable that option while installing Pi-hole. Now, we need to configure our Wireguard client to use Pi-hole as a nameserver for DNS resolution. # on the client add the following line sudo vim /etc/wireguard/wg0.conf [Interface] ... DNS = ... Pi-hole runs on port 53 and accepts DNS queries over UDP. Any query is first checked by Pi-hole in the blacklist. If it’s present in the blacklist, it’s immediately dropped. If not, Pi-hole will forward our DNS query to one of the forwarder configured (1.1.1.1 for eg or our custom server, explained in the next step). Setting up Unbound DNS# I don’t mind trusting Cloudflare. But I simply don’t have to. :) And once you make peace with the fact that you don’t need 3rd party companies controlling your networking stack, you’ll sleep better. I have setup Unbound without any forwarders. Unbound is a recursive resolver which supports DNSSec and caching mainly. Unbound first checks if the query exists in cache and if it does, it directly returns the “answer”. Otherwise it talks to the root nameserver and then the whole DNS dance happens. Since our DNS query is now split into multiple parts, where each nameserver is only being queried for a part of the FQDN (also known as QNAME minimialistion), it becomes a lot harder for anyone to intercept or reconstruct your DNS queries. You can install unbound using $ sudo apt-get install unbound To start using Unbound, we need a file root.hints which contains information about root nameservers. You can cron this to fetch a new copy every once in 3-4 months, it hardly changes. $ wget -O root.hints https://www.internic.net/domain/named.root $ sudo mv root.hints /var/lib/unbound/ $ sudo service unbound restart You can verify if the DNS queries are being resolved by: # 6363 is where I have configured my Unbound server to listen dig mrkaran.dev @127.0.0.1 -p 6363 Pi-hole official docs have a great explainer on how to configure Unbound with Pi-hole so I won’t be repeating the steps here again. You can configure Pi-hole to forward accepted DNS queries from port 53 (standard) to 127.0.0.1#6363 (unbound). The Endgame# I plan to self-host a couple of more things. DNS is something I am really interested in and in future I plan to host my own DNScrypt server soon-ish. I believe if you own your data you’re in better control of your digital identity. Watching too much of Black Mirror added to the paranoia to an extent, I suppose! I grew up in the late nineties and I’ve seen the internet primarily as a set of the decentralized toolchain. There’s no reason we should let go any of that to the hands of a few corp giants and make it centralized. Ending this long-ish post by a beautiful quote: Study after study has shown that human behaviour changes when we know we’re being watched. Under observation, we act less free, which means we effectively are less free. ― Edward Snowden Cheers! :)]]>

If I have nothing to hide then I have nothing to show either