DATA MINING WITH SUPERALGORES

Old pots produce the best soup.

Science has always inspired indicator design. From physics to signal processing, many indicators use concepts from mechanical engineering, electronics, and probability. In Superalgos' Data Mining section, we've explored using thermodynamics and information theory to construct indicators and using statistical and probabilistic techniques like reduced normal law to take advantage of low probability events.

An asset's price is like a mechanical object revolving around its moving average. Using this approach, we could design an indicator using the oscillator's Total Energy. An oscillator's energy is finite and constant. Since we don't expect the price to follow the harmonic oscillator, this energy should deviate from the perfect situation, and the maximum of divergence may provide us valuable information on the price's moving average.

Definition of the Harmonic Oscillator in Few Words

Sinusoidal function describes a harmonic oscillator. The time-constant energy equation for a harmonic oscillator is:

With

Time saves energy.

In a mechanical harmonic oscillator, total energy equals kinetic energy plus potential energy. The formula for energy is the same for every kind of harmonic oscillator; only the terms of total energy must be adapted to fit the relevant units. Each oscillator has a velocity component (kinetic energy) and a position to equilibrium component (potential energy).

The Price Oscillator and the Energy Formula

Considering the harmonic oscillator definition, we must specify kinetic and potential components for our price oscillator. We define oscillator velocity as the rate of change and equilibrium position as the price's distance from its moving average.

Price kinetic energy:

It's like:

With

and

L is the number of periods for the rate of change calculation and P for the close price EMA calculation.

Total price oscillator energy =

Given that an asset's price can theoretically vary at a limitless speed and be endlessly far from its moving average, we don't expect this formula's outcome to be constrained. We'll normalize it using Z-Score for convenience of usage and readability, which also allows probabilistic interpretation.

Over 20 periods, we'll calculate E's moving average and standard deviation.

We calculated Z on BTC/USDT with L = 10 and P = 21 using Knime Analytics.

The graph is detrended. We added two horizontal lines at +/- 1.6 to construct a 94.5% probability zone based on reduced normal law tables. Price cycles to its moving average oscillate clearly. Red and green arrows illustrate where the oscillator crosses the top and lower limits, corresponding to the maximum/minimum price oscillation. Since the results seem noisy, we may apply a non-lagging low-pass or multipole filter like Butterworth or Laguerre filters and employ dynamic bands at a multiple of Z's standard deviation instead of fixed levels.

Kinetic Detrender Implementation in Superalgos

The Superalgos Kinetic detrender features fixed upper and lower levels and dynamic volatility bands.

The code is pretty basic and does not require a huge amount of code lines.

It starts with the standard definitions of the candle pointer and the constant declaration :

let candle = record.current
let len = 10
let P = 21
let T = 20
let up = 1.6
let low = 1.6

Upper and lower dynamic volatility band constants are up and low.

We proceed to the initialization of the previous value for EMA :

if (variable.prevEMA === undefined) {
    variable.prevEMA = candle.close
}

And the calculation of EMA with a function (it is worth noticing the function is declared at the end of the code snippet in Superalgos) :

variable.ema = calculateEMA(P, candle.close, variable.prevEMA)
//EMA calculation
function calculateEMA(periods, price, previousEMA) {
    let k = 2 / (periods + 1)
    return price * k + previousEMA * (1 - k)
}

The rate of change is calculated by first storing the right amount of close price values and proceeding to the calculation by dividing the current close price by the first member of the close price array:

variable.allClose.push(candle.close)
if (variable.allClose.length > len) {
    variable.allClose.splice(0, 1)
}
if (variable.allClose.length === len) {
    variable.roc = candle.close / variable.allClose[0]
} else {
    variable.roc = 1
}

Finally, we get energy with a single line:

variable.E = 1 / 2 * len * variable.roc + 1 / 2 * P * candle.close / variable.ema

The Z calculation reuses code from Z-Normalization-based indicators:

variable.allE.push(variable.E)
if (variable.allE.length > T) {
    variable.allE.splice(0, 1)
}
variable.sum = 0
variable.SQ = 0
if (variable.allE.length === T) {
    for (var i = 0; i < T; i++) {
        variable.sum += variable.allE[i]
    }
    variable.MA = variable.sum / T
for (var i = 0; i < T; i++) {
        variable.SQ += Math.pow(variable.allE[i] - variable.MA, 2)
    }
    variable.sigma = Math.sqrt(variable.SQ / T)
variable.Z = (variable.E - variable.MA) / variable.sigma
} else {
    variable.Z = 0
}
variable.allZ.push(variable.Z)
if (variable.allZ.length > T) {
    variable.allZ.splice(0, 1)
}
variable.sum = 0
variable.SQ = 0
if (variable.allZ.length === T) {
    for (var i = 0; i < T; i++) {
        variable.sum += variable.allZ[i]
    }
    variable.MAZ = variable.sum / T
for (var i = 0; i < T; i++) {
        variable.SQ += Math.pow(variable.allZ[i] - variable.MAZ, 2)
    }
    variable.sigZ = Math.sqrt(variable.SQ / T)
} else {
    variable.MAZ = variable.Z
    variable.sigZ = variable.MAZ * 0.02
}
variable.upper = variable.MAZ + up * variable.sigZ
variable.lower = variable.MAZ - low * variable.sigZ

We also update the EMA value.

variable.prevEMA = variable.EMA

Conclusion

We showed how to build a detrended oscillator using simple harmonic oscillator theory. Kinetic detrender's main line oscillates between 2 fixed levels framing 95% of the values and 2 dynamic levels, leading to auto-adaptive mean reversion zones.

Superalgos' Normalized Momentum data mine has the Kinetic detrender indication.

All the material here can be reused and integrated freely by linking to this article and Superalgos.

This post is informative and not financial advice. Seek expert counsel before trading. Risk using this material.

Google announced 6% layoffs recently (or 12,000 people). This aligns it with most tech companies. A publicly contrite CEO explained that they had overhired during the COVID-19 pandemic boom and had to address it, but they were sorry and took full responsibility. I thought this was "bullshit" too. Meta, Amazon, Microsoft, and others must feel similarly. I spent 10 years at Google, and these things don't reflect well on the company's leaders.

All publicly listed companies have a fiduciary duty to act in the best interests of their shareholders. Dodge vs. Ford Motor Company established this (1919). Henry Ford wanted to reduce shareholder payments to offer cheaper cars and better wages. Ford stated.

My ambition is to employ still more men, to spread the benefits of this industrial system to the greatest possible number, to help them build up their lives and their homes. To do this we are putting the greatest share of our profits back in the business.

The Dodge brothers, who owned 10% of Ford, opposed this and sued Ford for the payments to start their own company. They won, preventing Ford from raising prices or salaries. If you have a vocal group of shareholders with the resources to sue you, you must prove you are acting in their best interests. Companies prioritize shareholders. Giving activist investors a stick to threaten you almost enshrines short-term profit over long-term thinking.

This underpins Google's current issues. Institutional investors who can sue Google see it as a wasteful company they can exploit. That doesn't mean you have to maximize profits (thanks to those who pointed out my ignorance of US corporate law in the comments and on HN), but it allows pressure. I feel for those navigating this. This is about unrestrained capitalism.

When Google went public, Larry Page and Sergey Brin knew the risks and worked hard to keep control. In their Founders' Letter to investors, they tried to set expectations for the company's operations.

Our long-term focus as a private company has paid off. Public companies do the same. We believe outside pressures lead companies to sacrifice long-term opportunities to meet quarterly market expectations.

The company has transformed since that letter. The company has nearly 200,000 full-time employees and a trillion-dollar market cap. Large investors have bought company stock because it has been a good long-term bet. Why are they restless now?

Other big tech companies emerged and fought for top talent. This has caused rising compensation packages. Google has also grown rapidly (roughly 22,000 people hired to the end of 2022). At $300,000 median compensation, those 22,000 people added $6.6 billion in salary overheads in 2022. Exorbitant. If the company still makes $16 billion every quarter, maybe not. Investors wonder if this value has returned.

Investors are right. Google uses people wastefully. However, by bluntly reducing headcount, they're not addressing the root causes and hurting themselves. No studies show that downsizing this way boosts productivity. There is plenty of evidence that they'll lose out because people will be risk-averse and distrust their leadership.

The company's approach also stinks. Finding out that you no longer have a job because you can’t log in anymore (sometimes in cases where someone is on call for protecting your production systems) is no way to fire anyone. Being with a narcissistic sociopath is like being abused. First, you receive praise and fancy perks for making the cut. You're fired by text and ghosted. You're told to appreciate the generous severance package. This firing will devastate managers and teams. This type of firing will take years to recover self-esteem. Senior management contributed to this. They chose the expedient answer, possibly by convincing themselves they were managing risk and taking the Macbeth approach of “If it were done when ’tis done, then ’twere well It were done quickly”.

Recap. Google's leadership did a stupid thing—mass firing—in a stupid way. How do we get rid of enough people to make investors happier? and "have 6% less people." Empathetic leaders should not emulate Elon Musk. There is no humane way to fire 12,000 people, but there are better ways. Why is Google so wasteful?

Ambition answers this. There aren't enough VP positions for a group of highly motivated, ambitious, and (increasingly) ruthless people. I’ve loitered around the edges of this world and a large part of my value was to insulate my teams from ever having to experience it. It’s like Game of Thrones played out through email and calendar and over video call.

Your company must look a certain way to be promoted to director or higher. You need the right people at the right levels under you. Long-term, growing your people will naturally happen if you're working on important things. This takes time, and you're never more than 6–18 months from a reorg that could start you over. Ambitious people also tend to be impatient. So, what do you do?

Hiring and vanity projects. To shape your company, you hire at the right levels. You value vanity metrics like active users over product utility. Your promo candidates get through by subverting the promotion process. In your quest for growth, you avoid performance managing people out. You avoid confronting toxic peers because you need their support for promotion. Your cargo cult gets you there.

Its ease makes Google wasteful. Since they don't face market forces, the employees don't see it as a business. Why would you do when the ads business is so profitable? Complacency causes senior leaders to prioritize their own interests. Empires collapse. Personal ambition often trumped doing the right thing for users, the business, or employees. Leadership's ambition over business is the root cause. Vanity metrics, mass hiring, and vague promises have promoted people to VP. Google goes above and beyond to protect senior leaders.

The decision-makers and beneficiaries are not the layoffees. Stock price increase beneficiaries. The people who will post on LinkedIn how it is about misjudging the market and how they’re so sorry and take full responsibility. While accumulating wealth, the dark room dwellers decide who stays and who goes. The billionaire investors. Google should start by addressing its bloated senior management, but — as they say — turkeys don't vote for Christmas. It should examine its wastefulness and make tough choices to fix it. A 6% cut is a blunt tool that admits you're not running your business properly. why aren’t the people running the business the ones shortly to be entering the job market?

This won't fix Google's wastefulness. The executives may never regain trust after their approach. Suppressed creativity. Business won't improve. Google will have lost its founding vision and us all. Large investors know they can force Google's CEO to yield. The rich will get richer and rationalize leaving 12,000 people behind. Cycles repeat.

It doesn’t have to be this way. In 2013, Nintendo's CEO said he wouldn't fire anyone for shareholders. Switch debuted in 2017. Nintendo's stock has increased by nearly five times, or 19% a year (including the drop most of the stock market experienced last year). Google wasted 12,000 talented people. To please rich people.

Product management wizardry.

Early adopters buy a product even if it hasn't hit the market or has flaws.

Who are the early adopters?

Early adopters try a new technology or product first. Early adopters are interested in trying or buying new technologies and products before others. They're risk-tolerant and can provide initial cash flow and product reviews. They help a company's new product or technology gain social proof.

Early adopters are most common in the technology industry, but they're in every industry. They don't follow the crowd. They seek innovation and report product flaws before mass production. If the product works well, the first users become loyal customers, and colleagues value their opinion.

What to do with early adopters?

They can be used to collect feedback and initial product promotion, first sales, and product value validation.

How to find early followers?

Start with your immediate environment and target audience. Communicate with them to see if they're interested in your value proposition.

1) Innovators (2.5% of the population) are risk-takers seeking novelty. These people are the first to buy new and trendy items and drive social innovation. However, these people are usually elite;

Early adopters (13.5%) are inclined to accept innovations but are more cautious than innovators; they start using novelties when innovators or famous people do;

3) The early majority (34%) is conservative; they start using new products when many people have mastered them. When the early majority accepted the innovation, it became ingrained in people's minds.

4) Attracting 34% of the population later means the novelty has become a mass-market product. Innovators are using newer products;

5) Laggards (16%) are the most conservative, usually elderly people who use the same products.

Stages of new information acceptance

1. The information is strange and rejected by most. Accepted only by innovators;

2. When early adopters join, more people believe it's not so bad; when a critical mass is reached, the novelty becomes fashionable and most people use it.

3. Fascination with a novelty peaks, then declines; the majority and laggards start using it later; novelty becomes obsolete; innovators master something new.

Problems with early implementation

Early adopter sales have disadvantages.

Higher risk of defects

Selling to first-time users increases the risk of defects. Early adopters are often influential, so this can affect the brand's and its products' long-term perception.

Not what was expected

First-time buyers may be disappointed by the product. Marketing messages can mislead consumers, and if the first users believe the company misrepresented the product, this will affect future sales.

Compatibility issues

Some technological advances cause compatibility issues. Consumers may be disappointed if new technology is incompatible with their electronics.

Method 5 WHY

Let's talk about 5 why, a good tool for finding project problems' root causes. This method is also known as the five why rule, method, or questions.

The 5 why technique came from Toyota's lean manufacturing and helps quickly determine a problem's root cause.

On one, two, and three, you simply do this:

1. We identify and frame the issue for which a solution is sought.

2. We frequently ponder this question. The first 2-3 responses are frequently very dull, making you want to give up on this pointless exercise. However, after that, things get interesting. And occasionally it's so fascinating that you question whether you really needed to know.

3. We consider the final response, ponder it, and choose a course of action.

Always do the 5 whys with the customer or team to have a reasonable discussion and better understand what's happening.

And the “five whys” is a wonderful and simplest tool for introspection. With the accumulated practice, it is used almost automatically in any situation like “I can’t force myself to work, the mood is bad in the morning” or “why did I decide that I have no life without this food processor for 20,000 rubles, which will take half of my rather big kitchen.”

An illustration of the five whys

A simple, but real example from my work practice that I think is very indicative, given the participants' low IT skills.  Anonymized, of course.

Users spend too long looking for tender documents.

Why? Because they must search through many company tender documents.

Why? Because the system can't filter department-specific bids.

Why? Because our contract management system requirements didn't include a department-tender link. That's it, right? We'll add a filter and be happy. but still…

why? Because we based the system's requirements on regulations for working with paper tender documents (when they still had envelopes and autopsies), not electronic ones, and there was no search mechanism.

Why? We didn't consider how our work would change when switching from paper to electronic tenders when drafting the requirements.

Now I know what to do in the future. We add a filter, enter department data, and teach users to use it. This is tactical, but strategically we review the same forgotten requirements to make all the necessary changes in a package, plus we include it in the checklist for the acceptance of final requirements for the future.

Errors when using 5 why

Five whys seems simple, but it can be misused.

Popular ones:

1. The accusation of everyone and everything is then introduced. After all, the 5 why method focuses on identifying the underlying causes rather than criticizing others. As a result, at the third step, it is not a good idea to conclude that the system is ineffective because users are stupid and that we can therefore do nothing about it.

2. to fight with all my might so that the outcome would be exactly 5 reasons, neither more nor less. 5 questions is a typical number (it sounds nice, yes), but there could be 3 or 7 in actuality.

3. Do not capture in-between responses. It is difficult to overestimate the power of the written or printed word, so the result is so-so when the focus is lost. That's it, I suppose. Simple, quick, and brilliant, like other project management tools.

Conclusion

Today we analyzed important study elements:

Early adopters and 5 WHY We've analyzed cases and live examples of how these methods help with product research and growth point identification. Next, consider the HADI cycle.