18-month follow-up on my self-concept work

About eighteen months ago, I found Steve Andreas’s book Transforming Your Self, and applied its techniques to fixing a number of issues in my self-concepts which had contributed to my depression and anxiety. Six weeks after those changes, I posted a report called “How I found & fixed the root problem behind my depression and anxiety after 20+ years”. I figured that by now it would be time for a follow-up on how those effects have lasted.

Overall summary and general considerations 

Looking back, this was definitely a major milestone in improving my mental health. I feel like since 2014, I have been ongoing a process of completely transforming myself from the depression- and anxiety-ridden person who was convinced that he had no other option than becoming a total failure, to someone calmly confident who has the option of constructing his life to his taste. I don’t claim to be there yet, but I feel like I’m constantly getting closer. I feel like the self-concept work discussed in my post, was one of the largest engines powering this transition. (Other major ones being me getting antidepressants, changing how I thought about ethics, and learning a new mindset from CFAR in 2014, properly learning Focusing and Core Transformation as well as starting to meditate according to The Mind Illuminated system in 2017, and starting to apply Internal Family Systems this year.)

There are two difficulties evaluating my self-concept post afterwards. First is that I have a poor emotional memory, so it’s a little hard for me to remember what I felt before these changes. The second is that after doing self-concept work, I’ve also done plenty of other things, such as meditation and moving together with some housemates, which have also had a definite impact on my mental health. I can’t know how well the self-concept work would have stuck around, if I hadn’t also implemented those other changes. It’s possible and even likely that some of my current results are because of those other changes instead.

At the same time, the self-concept work is also not independent from everything that I’ve done later. For instance, I think that being able to eliminate the feelings of pointless shame has been a major reason why I’ve been able to live with housemates and find them a definite net positive. Previously the feelings of shame would have made it too draining to have to engage in social interaction in my home on a regular basis, whereas now social interaction has tended to be much more energizing than it did in the past. But then again, there are also other skills which have made social fatigue less of an issue than sometime in the past, and which I’ve also been gradually training up.

But still, at least I can report on the various things in the post, and on how they’ve held up.

Things that seem to have been fixed for good

Generalized feelings of shame; being afraid of thinking that thoughts that might trigger feelings of shame; needing constant validation in order to avoid feelings of shame. I described the following in my post from last year:

I realized that I had a sense of unease, a vague feeling of shame… as if there was something shameful about me that I knew, but was trying to avoid thinking about. And I knew that I had felt this same vague shame many times before, often particularly when I was tired. […]

… there’s always an underlying insecurity, a sense of unease from the fact that anything might cause your attention to swing back to the [memories of being a terrible person]. You need a constant stream of external validation and evidence in order to keep your attention anchored on the examples [of being a good person]; the moment it ceases, your attention risks swinging to the [memories of being a terrible person] again.

As far as I can tell, this kind of thing simply doesn’t happen anymore. I still get feelings of guilt, if I have screwed up in some way, but there’s no shame or feeling of being a horrible person. Nor is there any need for external validation in this regard. I just know that I’m always doing the best that I can, and if I make a mistake that I need to learn from, then I feel the amount of guilt that’s necessary to motivate me to make amends and/or remember to act differently in the future. And that’s that.

Being motivated by a desire to prove to myself that I’m a good person. Previously I was trying to do a lot of things, but basically everything was strongly driven by a motivation to feel better about myself, and whenever it looked like something wasn’t likely to help with that goal, I would get demotivated. […] Previously when I was trying to do things to “save the world”, there was a strong component of doing it for the sake of guilt, feeling bad, or trying to win respect or status from others.

Basically fixed; this caused a period of readjustment, in that I had been doing things which had been optimized for looking good in the eyes of people that I admired, even when I personally hadn’t felt on a gut level that they made much sense. It took a while to readjust and find things which felt worth doing, but now I mostly feel like I’m doing things because they are genuinely derived from my values, rather than to avoid shame.

I still occasionally have something-like-guilt as a factor in thinking about what I want to do, but it mostly pops up when I notice that I’m not satisfying all of my own values and neglecting something that I actually care about. I’m no longer doing things “for the sake of guilt”, in the sense that I would do something and then keep feeling guilty regardless. If you find yourself regularly experiencing guilt, then you are using guilt incorrectly; in this respect as well, I’m using guilt much more correctly now.

Insecurity in relationships and with romantic partners; very detailed escapist romantic fantasies. If I was in a relationship, I would tend to very strongly highlight some qualities that I felt I had and which I felt bad about, in an attempt to get my partner to explicitly express being okay with them. […]

… much of my desire and need to be in a relationship was another way of trying to look for external validation, some kind of evidence that there was somebody who would accept me and would want to be with me. I used to have a lot of pretty detailed romantic fantasies; a lot of them lost their appeal after I fixed my self-concept.

Evaluating this is slightly harder since I haven’t actually been in a relationship since writing that post. However, judging from the way that I’ve felt towards and interacted with potential romantic partners as well as women I’ve been intimate friends with, and how I’ve felt about relationships in general, this feels basically fixed. Being single is far from my ideal preference, but it’s not particularly terrible either, and I don’t spend much time absorbed in detailed fantasies when I could be doing something else. I’m also much more comfortable with intimate friendships which are ambiguous about whether or not they might turn more romantic; I can be genuinely happy either way.

Mostly fixed, might still pop up a bit

Obsessive sexual fantasies. Without going into too much detail, previously my sexuality and fantasies had been very strongly entwined around a few paraphilias, which provided a great deal of emotional comfort. A lot of those fantasies were obsessive to the point of being bothersome.

At the time of writing my post, I reported that these basically disappeared. They remained gone for a while, but eventually some (not all) of them came back, though considerably transformed. They are fun to engage with occasionally, and they might get a bit bothersome if I think about them too much. But whenever they start getting that mildly obsessive flavor, it tends to act as a natural disincentive for me to continue thinking about them, and then they quiet down again.

Partially fixed, but with other causes as well

Feelings of anxiety and a need to escape. It feels that, large parts of the time, my mind is constantly looking for an escape, though I’m not entirely sure what exactly it is trying to escape from. But it wants to get away from the current situation, whatever the current situation happens to be. To become so engrossed in something that it forgets about everything else.

Unfortunately, this often leads to the opposite result. My mind wants that engrossment right now, and if it can’t get it, it will flinch away from whatever I’m doing and into whatever provides an immediate reward. Facebook, forums, IRC, whatever gives that quick dopamine burst. That means that I have difficulty getting into books, TV shows, computer games: if they don’t grab me right away, I’ll start growing restless and be unable to focus on them. Even more so with studies or work, which usually require an even longer “warm-up” period before one gets into flow.

This kind of a thing still happens; apparently the anxiety from poor self-concepts was only one of its causes. I now think that it’s more of an executive dysfunction symptom, in that various causes of stress or feeling bad can trigger a self-reinforcing loop of feeling bad, trying to escape that badness, feeling even more bad for failing to escape it, etc. My feelings of shame were definitely one cause, but many other things can also trigger it. Meditation and Focusing / IFS work have been a major aid in fixing several other causes.

Insecurities based on shame vs. instrumental considerations. Suppose that you have an unstable self-concept around “being a good person”, and you commit some kind of a faux pas. Or even if you haven’t actually committed one, you might just be generally unsure of whether others are getting a bad impression of you or not. Now, there are four levels on which you might feel bad about the real or imagined mistake:

  1. Feeling bad because you think you’re an intrinsically bad person
  2. Feeling bad because you suspect others think bad of you and that this is intrinsically bad (if other people think bad of you, that’s terrible, for its own sake)
  3. Feeling bad because you suspect others think bad of you and that this is instrumentally bad (other people thinking bad of you can be bad for various social reasons)
  4. Feeling bad because you might have hurt or upset someone, and you care about what others feel

Out of these, #3 and #4 are reasonable, #1 and #2 less so. When I fixed my self-concept, reaction #1 mostly vanished. But interestingly, reaction #2 stuck around for a while… or at least, a fear of #2 stuck around for a while.

#1 and #2 seem to indeed have disappeared; however, I’ve still continued to experience insecurities which have taken the forms of what seems like excessive worries of #3 and #4 (thinking that I’ve displeased someone in a way which will make them like me less, as well as worrying that someone might have felt upset over something that they in all likelihood won’t even remember). These seem to be the kinds of issues that can’t be fixed by internal work alone, since they are about the external world: in order to evaluate how justified these are, I need to actually test the extent to which something e.g. makes other people dislike me.

This work is still ongoing, but I’ve been making progress. Major contributors to current progress are the skills of integrating the cautions from my insecurities and tentatively considering emotional stories. These seem to have the effect that parts of my mind which have long held extreme beliefs about how cautious I should be, get listened to in a fairer way, causing them to update their beliefs to less extreme ones.

Difficulties in self-motivation. Besides being able to work at all, I’m also able to consistently work from home. This was often basically impossible: the impulse to escape was just too strong, and I needed to go elsewhere, preferably co-work with somebody else. Now I’ve cut down on co-working a lot, because leaving my home would take time, and I get more done if I don’t need to spend that time on travel.

This varies; implementing these fixes seems to have provided a temporary motivational boost allowing me to get a lot of work done with just the reward of financial security. When I find things to do that I’m significantly motivated by, then I seem to be able to work on them pretty well, even from home. However, anything that I’m not significantly motivated by still requires a lot of external structure for me to get anything done. Again, this seems like a manifestation of executive dysfunction issues more generally.

My initial motivation boost expired for a while, and I soon ran into new problems (I’ll discuss these below). It has taken a while to find promising new directions and figure out my new motivations so that I can do work more consistently, but (again thanks to meditation and Focusing / IFS work) in the last few months I’ve been starting to feel more consistently self-motivated.

In progress of being fixed after being made worse by the self-concept work

Lack of motivation once escaping the pain was no longer as motivating. For a while, there was a sense that my life had gotten more boring. Remember that analogy about being hungry all the time and focusing all your energies on food, and then being transformed into an android which didn’t need to eat? Your previous overriding priority of finding food being gone, you wouldn’t know what to do anymore. You’d feel okay, and it would be a steady okay – no lows, but also no particular highs.

The fixes in the post had the problem that I no longer felt actively bad; but eventually I started to notice that, having largely structured my life, habits and brain around escaping the badness, I didn’t have any particularly wholesome ways of feeling good. Even though I had fixed a major cause behind my depression and burnouts, they had still left pretty deep marks in my brain. After a while, I started to feel acutely anhedonic – limited in my ability to get pleasure from anything. The fact that many of my previous obsessive fantasies had been eliminated probably made this worse, since they had at least been a source of pleasure and motivation.

But this is still a good development. The goal of life isn’t to be free of problems; it’s to have more interesting problems, and this is definitely a much more interesting problem. I’ve been trying new things, from going to museums to generally being more open to stuff. I’m working on fixing the remaining mental blocks that are keeping me in place rather than experiencing stuff.

I’m gradually relearning to genuinely enjoy things. And that feels good: I feel like I’m just getting started in the process of rebuilding myself.

Can’t wait to see where I’ll be in a few year’s time.


Tentatively considering emotional stories (IFS and “getting into Self”)

I’ve recently been getting a lot out of the psychotherapy model of Internal Family Systems, as described in this book. I just wrote a comment on Slate Star Codex describing some of its basics and what I’ve gotten out of it, and thought that I might as well repost it here:

I recommend this book, though with the note that I often don’t need to follow the full process outlined there. Sometimes it’s definitely necessary, but what I’ve found even more commonly useful is something that it discusses at the beginning of the book, which it calls “getting into self”.

Here’s the basic idea. Suppose that a part of your mind is really angry at someone, and telling a story (which might not be true) about how that person is a horrible person with no redeeming qualities. Internal Family Systems says that there are three modes in which you might react to that part:

First, you may be entirely blended with it (for those familiar, this corresponds to what Acceptance and Commitment Therapy calls cognitive fusion). This means that you are experiencing everything in terms of the story that it is telling, and have forgotten that this is an emotional reaction. So you feel that it’s just objectively true that the other person is horrible and with no redeeming qualities.

Or you might be partially blended with it. In this case, you realize that you are experiencing an emotional reaction, and that your thoughts and feelings might not be entirely justified, but you still feel them and might not be able to stop yourself from behaving according to them anyway.

Finally, you might be “in Self”, meaning entirely unblended. Here you are still aware of the emotions and thoughts, but your subjective experience is that they’re not your emotions, they’re someone else’s – they’re coming from a part of your mind which is experienced as separate from “you”. In this mode, you do not feel threatened or overwhelmed by them, and you can maintain a state of open curiosity towards whether or not they are actually true.

My experience is that usually if I have an unpleasant emotion, I will try to do one of two things: either reject it entirely and push it out of my mind, or buy into the story that it’s telling and act accordingly. Once I learned the techniques for getting into Self, I got the ability to sort of… just hang out with the emotion, neither believing it to be absolutely true nor needing to show it to be false. And then if I e.g. had feelings of social anxiety, I could keep those feelings around and go into a social situation anyway, making a kind of mental move that I might describe as “yes, it’s possible that these people all secretly hate me; I’m going to accept that as a possibility without trying to add any caveats, but also without doing anything else than accepting its possibility”.

The consequence has been that this seems to make the parts of my mind with beliefs like “doing this perfectly innocuous thing will make other people upset” actually update their beliefs. I do the thing, the parts with this belief get to hang around and observe what happens, notice that nobody seems upset at me, and then they are somewhat less likely to bring up similar concerns in the future.

In terms of global workspace theory, my model here is that there’s a part of the mind that’s bringing up a concern that should be taken into account in decision-making. The concern may or may not be justified, so the correct thing to do is to consider its possibility, but not necessarily give it too much weight. Going into Self and letting the message stay in consciousness this way seems to make it available for decision-making, and often the module that’s bringing it up is happy to just have its message received and evaluated; you don’t have to do anything more than that, if it’s just holding it up as a tentative consideration to be evaluated.

The book has a few different techniques that you can use for getting into Self. One that I often use is to try to get a sense of where in my body the emotional sensations are coming from, and then let my mind create a visualization based on those. Once I have a visualization and a physical location of the part, it’s easier to experience it as “not me”. Another thing that I do is to just make that mental move that I described – “okay, this is a possibility, so I’m just going to test it out”. I find it useful to first stay blended with the part for a while, to get a sense of what exactly is the story that it’s trying to tell, before unblending and getting into Self.

E.g. a while back I was having a sense of loneliness as I laid down for a nap. I stepped into the part’s perspective to experience it for a while, then unblended; now I felt it as a black ice hockey puck levitating around my lower back. I didn’t really do anything other than let it be there, and maintained a connection with it. Gradually it started generating a pleasant warmth, and then the visualization transformed into a happy napping cartoon fox, curled up inside a fireball that it was using as its blanket. And then I was no longer feeling lonely.

That said, sometimes a part is not content to just raise a tentative possibility; sometimes it feels like something is an emergency, so you must act right away. Obviously, sometimes you really are in an emergency, so this is justified! But often times it’s based on the part having an unrealistic fear, which in the IFS model tends to be a result of some past trauma which it is reliving, not realizing that the circumstances of your life have changed and you’re now capable of dealing with it. In that case, you need to do the full process described in the book, where you basically get in proper contact with the part in question and address its concerns. (Actually it’s a bit more complicated than this, since the IFS model holds that there are many different kinds of parts that may have relationships with each other – so the “beer-drinking part” may be drinking beer in order to keep the traumatized part numb and safely out of consciousness, so you may actually need to deal with two different parts separately. The book goes into a lot more detail.)

Incorrect hypotheses point to correct observations

1. The Consciousness Researcher and Out-Of-Body Experiences

In his book Consciousness and the Brain, cognitive neuroscientist Stansilas Dehaene writes about scientifically investigating people’s reports of their out-of-body experiences:

the Swiss neurologist Olaf Blanke[ did a] beautiful series of experiments on out-of-body experiences. Surgery patients occasionally report leaving their bodies during anesthesia. They describe an irrepressible feeling of hovering at the ceiling and even looking down at their inert body from up there. […]

What kind of brain representation, Blanke asked, underlies our adoption of a specific point of view on the external world? How does the brain assess the body’s location? After investigating many neurological and surgery patients, Blanke discovered that a cortical region in the right temporoparietal junction, when impaired or electrically perturbed, repeatedly caused a sensation of out-of-body transportation. This region is situated in a high-level zone where multiple signals converge: those arising from vision; from the somatosensory and kinesthetic systems (our brain’s map of bodily touch, muscular, and action signals); and from the vestibular system (the biological inertial platform, located in our inner ear, which monitors our head movements). By piecing together these various clues, the brain generates an integrated representation of the body’s location relative to its environment. However, this process can go awry if the signals disagree or become ambiguous as a result of brain damage. Out-of-body flight “really” happens, then—it is a real physical event, but only in the patient’s brain and, as a result, in his subjective experience. The out-of-body state is, by and large, an exacerbated form of the dizziness that we all experience when our vision disagrees with our vestibular system, as on a rocking boat.

Blanke went on to show that any human can leave her body: he created just the right amount of stimulation, via synchronized but delocalized visual and touch signals, to elicit an out-of-body experience in the normal brain. Using a clever robot, he even managed to re-create the illusion in a magnetic resonance imager. And while the scanned person experienced the illusion, her brain lit up in the temporoparietal junction—very close to where the patient’s lesions were located.

We still do not know exactly how this region works to generate a feeling of self-location. Still, the amazing story of how the out-of-body state moved from parapsychological curiosity to mainstream neuroscience gives a message of hope. Even outlandish subjective phenomena can be traced back to their neural origins. The key is to treat such introspections with just the right amount of seriousness. They do not give direct insights into our brain’s inner mechanisms; rather, they constitute the raw material on which a solid science of consciousness can be properly founded.

The naive hypotheses that out-of-body experiences represented the spirit genuinely leaving the body, were incorrect. But they were still pointing to a real observation, namely that there are conditions which create a subjective experience of leaving the body. That observation could then be investigated through scientific means.

2. The Artist and the Criticism

In art circles, there’s a common piece of advice that goes along the lines of:

When people say that they don’t like something about your work, you should treat that as valid information.

When people say why they don’t like it or what you could do to fix it, you should treat that with some skepticism.

Outside the art context, if someone tells you that they’re pissed off with you as a person (or that you make them feel good), then that’s likely to be true; but the reason that they give may not be the true reason.

People have poor introspective access to the reasons why they like or dislike something; when they are asked for an explanation, they often literally fabricate their reasons. Their explanation is likely false, even though it’s still pointing to something in the work having made them dislike it.

3. The Traditionalist and the Anthropologist

The Scholar’s Stage blog post “Tradition is Smarter Than You Are“, quotes Joseph Henrich’s The Secret of Our Success which reports that many folk traditions, such as not eating particular fish during pregnancy, are adaptive: not eating that fish during pregnancy is good for the child, mother, or both. But the people in question often do not know why they follow that tradition:

We looked for a shared underlying mental model of why one would not eat these marine species during pregnancy or breastfeeding—a causal model or set of reasoned principles. Unlike the highly consistent answers on what not to eat and when, women’s responses to our why questions were all over the map. Many women simply said they did not know and clearly thought it was an odd question. Others said it was “custom.” Some did suggest that the consumption of at least some of the species might result in harmful effects to the fetus, but what precisely would happen to the fetus varied greatly, though a nontrivial segment of the women explained that babies would be born with rough skin if sharks were eaten and smelly joints if morays were eaten. Unlike most of our interview questions on this topic, the answers here had the flavor of post-hoc rationalization: “Since I’m being asked for a reason, there must be a reason, so I’ll think one up now.” This is extremely common in ethnographic fieldwork, and I’ve personally experienced it in the Peruvian Amazon with the Matsigenka and with the Mapuche in southern Chile.

The people’s hypotheses for why they do something is wrong. But their behavior is still pointing to the fish in question being bad to eat during pregnancy.

4. The Martial Artist and the Ki

In Types of Knowing, Valentine writes:

Another example is the “unbendable arm” in martial arts. I learned this as a matter of “extending ki“: if you let magical life-energy blast out your fingertips, then your arm becomes hard to bend much like it’s hard to bend a hose with water blasting out of it. This is obviously not what’s really happening, but thinking this way often gets people to be able to do it after a few cumulative hours of practice.

But you know what helps better?

Knowing the physics.

Turns out that the unbendable arm is a leverage trick: if you treat the upward pressure on the wrist as a fulcrum and you push your hand down (or rather, raise your elbow a bit), you can redirect that force and the force that’s downward on your elbow into each other. Then you don’t need to be strong relative to how hard your partner is pushing on your elbow; you just need to be strong enough to redirect the forces into each other.

Knowing this, I can teach someone to pretty reliably do the unbendable arm in under ten minutes. No mystical philosophy needed.

The explanation about magical life energy was false, but it was still pointing to a useful trick that could be learned and put to good use.

Observations and the hypotheses developed to explain them often get wrapped up, causing us to evaluate both as a whole. In some cases, we only hear the hypothesis rather than the observation which prompted it. But people usually don’t pull their hypotheses out of entirely thin air; even an incorrect hypothesis is usually entangled with some correct observations. If we can isolate the observation that prompted the hypothesis, then we can treat the hypothesis as a burdensome detail to be evaluated on its own merits, separate from the original observation. At the very least, the existence of an incorrect but common hypothesis suggests to us that there’s something going on that needs to be explained.

Mark Eichenlaub: How to develop scientific intuition

Recently on the CFAR alumni mailing list, someone asked a question about how to develop scientific intuition. In response, Mark Eichenlaub posted an excellent and extensive answer, which was so good that I asked for permission to repost it in public. He graciously gave permission, so I’ve reproduced his message below. (He otherwise retains the rights to this, meaning that the standard CC license on my blog doesn’t apply to this post.)

From: Mark Eichenlaub
Date: Tue, Oct 23, 2018 at 9:34 AM
Subject: Re: [CFAR Alumni] Suggestions for developing scientific intuition

Sorry for the length, I recently finished a PhD on this topic. (After I wrote the answer kerspoon linked, I went to grad school to study the topic.) This is specifically about solving physics problems but hopefully speaks to intuition a bit more broadly in places.

I mostly think of intuition as the ability to quickly coordinate a large number of small heuristics. We know lots of small facts and patterns, and intuition is about matching the relevant ones onto the current situation. The little heuristics are often pretty local and small in scope.

For example, the other day I heard this physics problem:

You set up a trough with water in it. You hang just barely less than half of the trough off the edge of a table, so that it balances, but even a small force at the far end would make it tip over.

You put a boat in the trough at the end over the table. The trough remains balanced.

Then you slowly push the boat down to the other end of the trough, so that’s it’s in the part of the trough that hangs out from the table. What happens? (I.E. does the trough tip over?)

The answer is (rot13) Gur gebhtu qbrf abg gvc; vg erznvaf onynaprq (nf ybat nf gur zbirzrag bs gur obng vf fhssvpvragyl fybj fb gung rirelguvat erznvaf va rdhvyvoevhz).

I knew this “intuitively”, by which I mean I got it within a second or so of understanding the question, and without putting in conscious effort to thinking about it. (I wasn’t certain I was right until I had consciously thought it out, but I was reasonably confident within a second, and my intuition bore out.) I don’t think this was due to some sort of general intuition about problem solving, science, physics, mechanics, or even floating. It felt like I could solve the problem intuitively specifically because I had seen sufficiently-similar things that led me to the specific heuristic “a floating object spreads its weight out evenly over the bottom of the container it’s floating in.” Then I think of “having intuition” in physics as having maybe a thousand little rules like that and knowing when to call on which one.

For this particular heuristic, there is a classic problem asking what happens to the water level in a lake if you are in a boat with a rock, and you throw the rock into the water and it sinks to the bottom. One solution to that problem is that when the rock is on the bottom of the lake, it exerts more force on that part of the bottom of the lake than is exerted at other places. By contrast, when the rock is still in the boat, the only thing touching the bottom of the lake is water, and the water pressure is the same everywhere, so the weight of the rock is distributed evenly across the entire lake. The total force on the bottom of the lake doesn’t change between the two scenarios (because gravity pulls on everything just as hard either way), when the rock is sitting on the bottom of the lake and the force on the bottom of the lake is higher under the rock, it must be lower everywhere else to compensate. The pressure everywhere else is $\rho g h$, so if that goes down, the level of the lake goes down. Conclusion: when you throw the rock overboard, the level of the lake goes down a bit. When I thought about that problem, I presumably built the “weight distributed evenly” heuristic. All I had to do was quickly apply it to the trough problem to solve that one as well.

And if someone else also had a background in physics but didn’t find the trough problem easy, it’s probably because they simply hadn’t happened to think about the boat problem, or some other similar problems, in the right way, and hadn’t come away with the heuristic about the weight of floating things being spread out evenly.

To me, this picture of intuition as small heuristics doesn’t look good for the idea of developing powerful intuition. The “weight gets spread out by floating” heuristic is not likely to transfer to much else. I’ve used it for two physics problems about floating things and, as far as I know, nothing else.

You can probably think of lots of similar heuristics. For example, “conservation of expected evidence“. You might catch a mistake in someone’s reasoning, or an error in a long probability calculation you made, if you happen to notice that the argument or calculation violates conservation of expected evidence. The nice thing about this is that it can happen almost automatically. You don’t have to stop after every calculation or argument and think, “does this break conservation of expected evidence?”. Instead, you wind up learning some sorts of triggers that you associate with the principle that prime it in your mind, and then, if it becomes relevant to the argument, you notice that and cite the principle.

In this picture, building intuition is about learning a large number of these heuristics, along with their triggers.

However, while the individual small heuristics are often the easiest things to point to in an intuitive solution to a problem, I do think there are more general, and therefore more transferrable parts of intuition as well. I imagine that the paragraph I wrote explaining the solution to the boat problem will be largely incomprehensible to someone who hasn’t studied physics. That’s partially because it uses concepts they won’t have a rigorous understanding of (e.g. pressure), that it tacitly uses small heuristics it didn’t explain (e.g. that the reason the pressure is the same along the bottom of the lake is that if it weren’t, there would be horizontal forces that push the water around until the pressure did equalize in this way), partially that it made simplifications that it didn’t state and it might not be clear are justified (e.g. that the bottom of the lake is flat). More importantly, it relies on a general framework of Newtonian mechanics. For example, there are a number of tacit applications of Newton’s laws in the argument. For example, I stated that the total force on the bottom of the lake is the same whether the rock is resting on the bottom or floating in the boat “because gravity pulls on everything just as hard either way”, but these aren’t directly connected concepts. Gravity pulls the system (boat + water + rock) down just as hard no matter where the rock is. That system is not accelerating, so by Newton’s second law, the bottom of the lake pushes up on that system just as hard in each scenario. And by Newton’s third law, the system pushes down on the bottom of the lake just as hard in each scenario. So understanding the argument involves some fairly general heuristics such as “apply Newton’s second law to an object in equilibrium to show that two forces on it have equal magnitude” – a heuristic I’ve used hundreds of times, and “decide what objects to define as part of a system fluidly as you go through a problem” (in this case, switching from thinking about the rock as a system to thinking about rock+boat+water as a single system) – a skill I’ve used hundreds to thousands of times across all of physics. (My job is to teach high schoolers to be really good at solving problems like this, so I spend way more time on it than most people, so applying a heuristic specific to solving introductory physics problems in a thousand independent instances is realistic for me.)

Then there may be more meta-level skills and heuristics that you develop in solving problems. These could be things like valuing non-calculation solutions, or believing that persevering on a tough problem is worthwhile. It’s also important that intuition isn’t just about having lots of little heuristics. It’s about organizing them and calling the right one up at the right time. You’ll have to ask yourself the right sorts of questions to prompt yourself to find the right heuristics, and that’s probably a pretty general skill.

There is a fair amount of research on trying to understand what all these little heuristics are and how to develop them, but I’m mostly familiar with the research in physics.

In the Quora answer kerspoon linked, I cited George Lakoff, and I still that he’s a good source for understanding how we go about taking primitive sorts of concepts (e.g. “up” and “down”) and using and adapting them, via partial metaphor, to understanding more abstract things. For a specific example that’s well-argued, see:

Wittmann, Michael C., and Katrina E. Black. “Mathematical actions as procedural resources: An example from the separation of variables.” Physical Review Special Topics-Physics Education Research 11.2 (2015): 020114.

They argue that students understand the arithmetic action “separation of variables” via analogy to their physical understanding of taking things and physically moving them around. However, I think Wittman and Black’s work is incomplete. For example, it doesn’t explain why students using the motion analogy for separation of variables do it correctly – they could just as well use motion to encode algebraically-invalid rules. Also, they don’t explain how the analogy develops. They just catalog that it exists.

A foundational work in trying to understand the components of physical intuition is:

DiSessa, Andrea A. “Toward an epistemology of physics.” Cognition and instruction 10.2-3 (1993): 105-225.

This work establishes “phenomenological primitives”; little core heuristics such as “near is more”, which are templates for physical reasoning. Drawing from these templates, we might conclude that the nearer you are to a speaker, the louder the sound, or that the nearer you are to the sun, the hotter it will be (and therefore that summer is hot because the Earth is nearer the sun – a false but common and reasonable belief).

That’s a long and somewhat-obscure paper. I really like his student’s work

Sherin, Bruce L. “How students understand physics equations.” Cognition and instruction 19.4 (2001): 479-541.

Like Disessa, Sherin builds his own framework for what intuition is. His scope is more limited though, focusing solely on building and interpreting certain types of equations in a manner that combines “intuitive” physical ideas and mathematical templates. He spells this out in detail more in the paper, and it’s incredibly clear and well-argued. Probably my favorite paper in the field.

A more general reference that’s much more accessible than Disessa and more general an overview of cognition in physics than Sherin is
“How Should We Think About How Our Students Think” by my advisor, Joe Redish http://media.physics.harvard.edu/video/?id=COLLOQ_REDISH_093013 (video) https://arxiv.org/abs/1308.3911 (paper).

The actual process of building new heuristics is also studied, but over all I don’t think we know all that much. See my friend Ben’s paper

Dreyfus, Benjamin W., Ayush Gupta, and Edward F. Redish. “Applying conceptual blending to model coordinated use of multiple ontological metaphors.” International Journal of Science Education 37.5-6 (2015): 812-838.

for an example of theory-building around how we create new intuitions. He calls on a framework from cognitive science called “conceptual blending” that is rather formal, but I think pretty entertaining to read.

A relevant search terms in the education literature:

“conceptual change”

but I find a lot of this literature to be hard-to-follow and not always a productive use of time to read.

On the applied side, I think the state of the art in evidence-backed approaches to building intuition, at least in physics, is modeling instruction. I’m not sure what the best introduction to modeling instruction is. They have a website that seems okay. Eric Brewe writes on it and he’s usually very good. The basic idea is to have students collaboratively participate in the building of the theories of physics they’re using (in a specific way, with guidance and direction from a trained instructor), which gets them to think about the “whys” involved with a particular theory or model in a way they usually wouldn’t.

I have written some about why I think things like checking the extreme cases of a formula are powerful intuition-building tools. A preprint is available here: https://arxiv.org/pdf/1804.01639.pdf

However, I think it’s dangerous to have rules like “always check the dimensions of your answer”, “always check the extreme cases of a formula”, or even “always check that the numbers come out reasonable.” The reason is that having these things as procedures tends to encourage students to follow them by rote. A large part of the cognitive work involved isn’t in checking the extreme cases or the dimensions, but in realizing that in this particular situation, that would be a good thing to do. If you’re doing it only because an external prompt is telling you to, you aren’t building the appropriate meta-level habits. See https://www.tandfonline.com/doi/abs/10.1080/09500693.2017.1308037 for an example of this effect.

See papers on “metarepresentation” by Disessa and/or Sherin for another example of generalizable skills related to intuition and problem solving.

Unfortunately, I don’t think writing books well or writing courses of individual study is something we know much about. I don’t know anyone who has a significant grant for that; the most I’ve ever seen on it is a poster here or there at a conference. Generally, grants are awarded for improving high school and college courses, or for professional development programs, supporting department or institution level changes at schools, etc. So adults who just want to learn on their own are not really served much by the research on the area. If you’re an adult who wants to self-study theoretical physics with an eye towards intuition, I recommend Leonard Susskind’s series of courses “The Theoretical Minimum” (the first three courses exist as books, the rest only as video lectures). He approaches mathematical topics with what I find an intuitive approach in most cases. Of course the Feynman lectures on physics are also very good.

I’ll be building an introduction to physics course at Art of Problem Solving, starting work sometime this winter. It might be available in the spring, although students will mostly be middle and high school students (but anyone is welcome to take our courses). I currently teach an advanced physics problem-solving course at AoPS called “PhysicsWOOT”. I try to support intuition-building practices there, but the main aim is in training these many small heuristics which students need to solve contest problems.

There should be something like modeling instruction for adult independent learners, but I don’t know of it.