Looks like I have solution:

private void makeCollapsingToolbarLayoutLooksGood(CollapsingToolbarLayout collapsingToolbarLayout) {
try {
final Field field = collapsingToolbarLayout.getClass().getDeclaredField("mCollapsingTextHelper");
field.setAccessible(true);


final Object object = field.get(collapsingToolbarLayout);
final Field tpf = object.getClass().getDeclaredField("mTextPaint");
tpf.setAccessible(true);


((TextPaint) tpf.get(object)).setTypeface(Typeface.createFromAsset(getAssets(), "Roboto-Bold.ttf"));
((TextPaint) tpf.get(object)).setColor(getResources().getColor(R.color.ice));
} catch (Exception ignored) {
}
}

You can do something like this:

mCollapsingToolbarLayout.setTitle(getTitle());
mCollapsingToolbarLayout.setExpandedTitleTextAppearance(R.style.ExpandedAppBar);
mCollapsingToolbarLayout.setCollapsedTitleTextAppearance(R.style.CollapsedAppBar);

Corresponding textview style could be:

<style name="ExpandedAppBar" parent="@android:style/TextAppearance.Medium">
<item name="android:textSize">28sp</item>
<item name="android:textColor">#000</item>
<item name="android:textStyle">bold</item>
</style>


<style name="CollapsedAppBar" parent="@android:style/TextAppearance.Medium">
<item name="android:textSize">24sp</item>
<item name="android:textColor">@color/white</item>
<item name="android:textStyle">normal</item>
</style>

also see here for reference.

Update

Before we dive into the code let's first decide the textSize for our CollapsingToolbarLayout. Google published a website called material.io, this website also explains the best way on how to deal with Typography.

The article mentioned about "Heading" category which also explains the recommended font size to use in sp.

Although the article never mentioned the recommended CollapsingToolbarLayout's expanded size but the library com.android.support:design provides a TextAppearance for our case. With some digging with the source it turns out that that the size is 34sp (not mentioned in the article).

So how about the collapsed size? Luckily the article mentioned something and it is 20sp.

And the best TextAppearance so far that fits in collpased mode is TextAppearance.AppCompat.Title while our expanded mode TextAppearance is TextAppearance.Design.CollapsingToolbar.Expanded.

If you observe all our examples above all of them uses the REGULAR version of the font which is safe to say that Roboto regular will do the task unless you have specific requirements.

It might be too much work downloading the fonts itself why not use a library that has all the needed Roboto fonts? So I introduce a calligraphy library for Roboto e.g. Typer.

dependencies {
implementation 'com.android.support:design:28.0.0'
implementation 'com.rhexgomez.typer:typer-roboto:2.0.0'
}

<android.support.design.widget.CollapsingToolbarLayout
android:id="@+id/collapsing_toolbar"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:fitsSystemWindows="true"
app:contentScrim="?attr/colorPrimary"
app:expandedTitleMarginEnd="64dp"
app:expandedTitleMarginStart="48dp"
app:expandedTitleTextAppearance="@style/TextAppearance.Design.CollapsingToolbar.Expanded"
app:collapsedTitleTextAppearance="@style/TextAppearance.AppCompat.Title"
app:layout_scrollFlags="scroll|exitUntilCollapsed">

Java

// Java example
CollapsingToolbarLayout collapsingToolbar = findViewById(R.id.collapsing_toolbar);
collapsingToolbar.setCollapsedTitleTypeface(TyperRoboto.ROBOTO_REGULAR());
collapsingToolbar.setExpandedTitleTypeface(TyperRoboto.ROBOTO_REGULAR());

Kotlin

// Kotlin example
collapsing_toolbar.apply {
setCollapsedTitleTypeface(TyperRoboto.ROBOTO_REGULAR)
setExpandedTitleTypeface(TyperRoboto.ROBOTO_REGULAR)
}

_{This is a 2019 update because the Typer library is updated! I am also the author of the library.}

You can use the new public methods, on CollapsingToolbarLayout to set the typeface for the collapsed and expanded title, like so:

final Typeface tf = Typeface.createFromAsset(context.getAssets(), "fonts/FrutigerLTStd-Light.otf");
collapsingToolbar.setCollapsedTitleTypeface(tf);
collapsingToolbar.setExpandedTitleTypeface(tf);

This seems to have been added in the design support library 23.1.0, and is a very welcome addition.

    mCollapsingToolbar.setTitle(getTitle());
mCollapsingToolbar.setExpandedTitleTextAppearance(R.style.ExpandedAppBar);
mCollapsingToolbar.setCollapsedTitleTextAppearance(R.style.CollapsedAppBar);


<style name="ExpandedAppBar" parent="@android:style/TextAppearance.Medium">
<item name="android:textSize">28sp</item>
<item name="android:textColor">#000</item>
<item name="android:textStyle">bold</item>
</style>


<style name="CollapsedAppBar" parent="@android:style/TextAppearance.Medium">
<item name="android:textSize">24sp</item>
<item name="android:textColor">@color/white</item>
<item name="android:textStyle">normal</item>
</style>


<style name="ExpandedAppBarPlus1" parent="@android:style/TextAppearance.Medium">
<item name="android:textSize">28.5sp</item>
<item name="android:textColor">#000</item>
<item name="android:textStyle">bold</item>
</style>


<style name="CollapsedAppBarPlus1" parent="@android:style/TextAppearance.Medium">
<item name="android:textSize">24.5sp</item>
<item name="android:textColor">@color/white</item>
<item name="android:textStyle">normal</item>
</style>

Reference here CollapsingToolbarLayout setTitle not work correctly

Change the font size or parent.

<style name="expandedappbar" parent="@android:style/TextAppearance.Medium"> //Change Medium to Small
<item name="android:textSize">28sp</item>  <!--Or Change the font size -->
<item name="android:textColor">@color/white</item>
<item name="android:textStyle">bold</item>
</style>


<style name="collapsedappbar" parent="@android:style/TextAppearance.Medium">
<item name="android:textSize">18sp</item>
<item name="android:textColor">@color/white</item>
</style>

Code is here

 <android.support.design.widget.CollapsingToolbarLayout
android:id="@+id/collapsingToolbarLayout"
android:layout_width="match_parent"
app:expandedTitleTextAppearance="@style/Toolbar.TitleText"
app:collapsedTitleTextAppearance="@style/Toolbar.TitleText"
android:layout_height="match_parent"
app:contentScrim="?attr/colorPrimary"
app:expandedTitleMarginStart="@dimen/expanded_toolbar_title_margin_start"
app:layout_scrollFlags="scroll|exitUntilCollapsed">

Add these code lines in CollapsingToolbarLayout layout

 app:expandedTitleTextAppearance="@style/Toolbar.TitleText"
app:collapsedTitleTextAppearance="@style/Toolbar.TitleText"

And the code which is given below, in style.xml

<style name="Toolbar.TitleText" parent="TextAppearance.Widget.AppCompat.Toolbar.Title">
<item name="android:textSize">16sp</item>
</style>

To add to all the answers here, It did not work for me in xml no matter where I tried to apply, in AppTheme, referencing in styles. I am currently using support library 27.1.1

It worked only programatically.

Typeface typeface = ResourcesCompat.getFont(this, R.font.my_custom_font);
collapsingToolbarLayout.setCollapsedTitleTypeface(typeface);
collapsingToolbarLayout.setExpandedTitleTypeface(typeface);

Android 8.0 (API level 26) have introduces a new feature, Fonts in XML

Now you can define fonts as resource https://developer.android.com/guide/topics/ui/look-and-feel/fonts-in-xml.html

• Put your fonts in in res-> font-> folder
• Define font.xml

   <?xml version="1.0" encoding="utf-8"?>
<font-family xmlns:android="http://schemas.android.com/apk/res/android">
<font
android:fontStyle="normal"
android:fontWeight="400"
android:font="@font/lobster_regular" />
<font
android:fontStyle="italic"
android:fontWeight="400"
android:font="@font/lobster_italic" />
</font-family>

next set

    val typeface = ResourcesCompat.getFont(this, R.font.lobster_regular)
htab_collapse_toolbar.setCollapsedTitleTypeface(typeface)
htab_collapse_toolbar.setExpandedTitleTypeface(typeface)