Yes, you should draw calibration curves without binning.
The calibration curve lives in a plane where the x-axis is the probability of the prediction and the y-axis is the actual proportion of outcomes. We can make each prediction live in this plane. A scored prediction is a pair of an outcome, b, either 0 or 1, and the earlier prediction p. The value p belongs on the x-axis. The value b is sort of like a value on the y-axis. Thus (p,b) makes sense on the x-y-plane. It is valuable to plot the scatterplot of this representation of the predictions. The calibration curve is a curve attempting to approximate this scatterplot. A technique for turning a scatter plot into the graph of a function is called a smoother. Every smoother yields a different notion of calibration curve. The most popular general-purpose smoother is loess and it is also the most popular smoother for the specific task of calibration curves without bins. Frank Harrell (2-28) suggests tweaking the algorithm, setting α=1.
Yes, you should draw calibration curves without binning.
The calibration curve lives in a plane where the x-axis is the probability of the prediction and the y-axis is the actual proportion of outcomes. We can make each prediction live in this plane. A scored prediction is a pair of an outcome, b, either 0 or 1, and the earlier prediction p. The value p belongs on the x-axis. The value b is sort of like a value on the y-axis. Thus (p,b) makes sense on the x-y-plane. It is valuable to plot the scatterplot of this representation of the predictions. The calibration curve is a curve attempting to approximate this scatterplot. A technique for turning a scatter plot into the graph of a function is called a smoother. Every smoother yields a different notion of calibration curve. The most popular general-purpose smoother is loess and it is also the most popular smoother for the specific task of calibration curves without bins. Frank Harrell (2-28) suggests tweaking the algorithm, setting α=1.